[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:09 dist-test-slave-kxl8 krb5kdc[398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:09 dist-test-slave-kxl8 krb5kdc[398](info): set up 2 sockets
Sep 25 16:38:09 dist-test-slave-kxl8 krb5kdc[398](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 16:38:10.062852 391 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:10.187067 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:41251
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:10.299878 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:10.300189 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:10.300282 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:10.304422 411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:10.304508 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:10.304546 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:10.304597 411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:10.304642 411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:10.310050 411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:10.311141 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:10.312186 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:10.321441 419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:10.321452 416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:10.321552 417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:10.321653 411 server_base.cc:1034] running on GCE node
I20260925 16:38:10.322299 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:10.323449 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:10.324632 411 hybrid_clock.cc:648] HybridClock initialized: now 1790354290324565 us; error 83 us; skew 500 ppm
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:10.327291 411 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:10.328158 411 webserver.cc:460] Webserver started at http://127.0.97.254:46449/ using document root <none> and password file <none>
I20260925 16:38:10.328698 411 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:10.328756 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:10.328922 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:10.330997 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "04e9d916379c4408837954173c21ef2b"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "3a96e6e09000319e0d22007b0c506141"
server_key_iv: "56cf98492459de5a2d8fc1e499d4b0a1"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.331498 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "04e9d916379c4408837954173c21ef2b"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "3a96e6e09000319e0d22007b0c506141"
server_key_iv: "56cf98492459de5a2d8fc1e499d4b0a1"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.335280 411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:38:10.337644 426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.338965 411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:10.339069 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "04e9d916379c4408837954173c21ef2b"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "3a96e6e09000319e0d22007b0c506141"
server_key_iv: "56cf98492459de5a2d8fc1e499d4b0a1"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.339159 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:10.365603 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:10.368460 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:10.368664 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:10.375852 411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:41251
I20260925 16:38:10.375874 478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:41251 every 8 connection(s)
I20260925 16:38:10.376878 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:10.380693 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:10.384017 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 411
I20260925 16:38:10.384186 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:10.385876 391 external_mini_cluster.cc:1428] Setting key 10bccccaba2a1bb427082a51267a4b6b
I20260925 16:38:10.387370 479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b: Bootstrap starting.
I20260925 16:38:10.390196 479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:10.390933 479 log.cc:824] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:10.392827 479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b: No bootstrap required, opened a new log
I20260925 16:38:10.396832 479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } }
I20260925 16:38:10.397080 479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:10.397167 479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e9d916379c4408837954173c21ef2b, State: Initialized, Role: FOLLOWER
I20260925 16:38:10.397679 479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } }
I20260925 16:38:10.397835 479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:10.397907 479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:10.398007 479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:10.398954 479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } }
I20260925 16:38:10.399309 479 leader_election.cc:302] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e9d916379c4408837954173c21ef2b; no voters:
I20260925 16:38:10.399631 479 leader_election.cc:288] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:10.399713 484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:10.399890 484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [term 1 LEADER]: Becoming Leader. State: Replica: 04e9d916379c4408837954173c21ef2b, State: Running, Role: LEADER
I20260925 16:38:10.400245 484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } }
I20260925 16:38:10.400909 479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:10.402004 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e9d916379c4408837954173c21ef2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } } }
I20260925 16:38:10.402159 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:10.402431 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e9d916379c4408837954173c21ef2b. Latest consensus state: current_term: 1 leader_uuid: "04e9d916379c4408837954173c21ef2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9d916379c4408837954173c21ef2b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41251 } } }
I20260925 16:38:10.402518 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:10.402549 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:10.405855 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:10.411813 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:10.389635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.97.254:41251)
0925 16:38:10.390127 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:10.390157 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:10.391745 (+ 1588us) server_negotiation.cc:407] Connection header received
0925 16:38:10.394985 (+ 3240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:10.395001 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:10.395305 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:10.395621 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:10.397857 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.398946 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.400379 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.400849 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.400944 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:10.406305 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:10.406311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:10.406326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:10.406354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:10.408100 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.408628 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.408634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.408639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.408751 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.409070 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.409074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.409075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.409591 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:10.409805 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:10.410466 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 16:38:10.410771 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20260925 16:38:10.414472 493 catalog_manager.cc:1347] Generated new cluster ID: baf1ccdd337b49d7ab1129e8d7794b6f
I20260925 16:38:10.414532 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:10.424871 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:10.425791 493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:10.431689 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e9d916379c4408837954173c21ef2b: Generated new TSK 0
I20260925 16:38:10.432302 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:10.464507 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:10.577668 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:10.577934 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:10.577978 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:10.582136 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:10.582206 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:10.582314 505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:10.587555 505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:10.588649 505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:10.589797 505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:10.599661 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:10.599751 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:10.599715 505 server_base.cc:1034] running on GCE node
W20260925 16:38:10.599951 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:10.600332 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:10.601006 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:10.602181 505 hybrid_clock.cc:648] HybridClock initialized: now 1790354290602155 us; error 47 us; skew 500 ppm
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:10.604961 505 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:10.605880 505 webserver.cc:460] Webserver started at http://127.0.97.193:41523/ using document root <none> and password file <none>
I20260925 16:38:10.606436 505 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:10.606524 505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:10.606755 505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:10.608815 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "0cd38b23f49a41f69652f562fc15ac3f"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "9b0839cd89bedc2ec687d510936cba5c"
server_key_iv: "cc62248c223e11a4ba3af5949a9e808e"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.609453 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "0cd38b23f49a41f69652f562fc15ac3f"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "9b0839cd89bedc2ec687d510936cba5c"
server_key_iv: "cc62248c223e11a4ba3af5949a9e808e"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.613351 505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 16:38:10.615796 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.617094 505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:38:10.617223 505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "0cd38b23f49a41f69652f562fc15ac3f"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "9b0839cd89bedc2ec687d510936cba5c"
server_key_iv: "cc62248c223e11a4ba3af5949a9e808e"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.617378 505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:10.636833 505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:10.640175 505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:10.640381 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:10.640988 505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:10.641875 505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:10.641944 505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.642012 505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:10.642151 505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.650848 505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:34373
I20260925 16:38:10.650923 633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:34373 every 8 connection(s)
I20260925 16:38:10.651778 505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:10.661033 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 505
I20260925 16:38:10.661175 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:10.661477 391 external_mini_cluster.cc:1428] Setting key b12213e7a394f604ecadff3ab9469076
I20260925 16:38:10.666064 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:10.653792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:55013 (local address 127.0.97.254:41251)
0925 16:38:10.653926 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:10.653930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:10.654872 (+ 942us) server_negotiation.cc:407] Connection header received
0925 16:38:10.656165 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:10.656168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:10.656213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:10.656293 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:10.657481 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.658050 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.659146 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.659390 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.659471 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:10.662182 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:10.662185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:10.662198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:10.662222 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:10.663683 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.664296 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.664299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.664300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.664372 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.664778 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.664784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.664788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.665037 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:10.665152 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:10.665661 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 16:38:10.665815 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:10.666743 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:10.654210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41251 (local address 127.0.97.193:55013)
0925 16:38:10.654721 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:10.654765 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:10.655952 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:10.656450 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:10.656457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:10.656809 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:10.657348 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:10.657369 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.658152 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.658155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:10.659032 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:10.659040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.659514 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.659517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:10.659650 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:10.660231 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:10.660247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:10.662009 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:10.663800 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:10.663809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:10.663828 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:10.664202 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:10.664456 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:10.664462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:10.664466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:10.664671 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:10.665146 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:10.665151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:10.665428 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 16:38:10.665586 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 16:38:10.665864 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260925 16:38:10.667943 634 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41251
I20260925 16:38:10.668186 634 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:10.668687 634 heartbeater.cc:511] Master 127.0.97.254:41251 requested a full tablet report, sending...
I20260925 16:38:10.670217 443 ts_manager.cc:194] Registered new tserver with Master: 0cd38b23f49a41f69652f562fc15ac3f (127.0.97.193:34373)
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:10.671439 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:55013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:10.689231 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:10.802028 639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:10.802270 639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:10.802345 639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:10.806514 639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:10.806576 639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:10.806667 639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:10.811834 639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:10.812868 639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:10.813902 639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:10.823354 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:10.823366 644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:10.823374 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:10.823490 639 server_base.cc:1034] running on GCE node
I20260925 16:38:10.824011 639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:10.824654 639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:10.825819 639 hybrid_clock.cc:648] HybridClock initialized: now 1790354290825807 us; error 29 us; skew 500 ppm
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:10.828387 639 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:10.829262 639 webserver.cc:460] Webserver started at http://127.0.97.194:43417/ using document root <none> and password file <none>
I20260925 16:38:10.829861 639 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:10.829944 639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:10.830169 639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:10.832165 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "a226bba587854e2e8390e75b31cd4a85"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "e182744e5fa500e702dc7ac73c57ce11"
server_key_iv: "ca0acf1a8bb265c6ed6d1beb4a44cd26"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.832738 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "a226bba587854e2e8390e75b31cd4a85"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "e182744e5fa500e702dc7ac73c57ce11"
server_key_iv: "ca0acf1a8bb265c6ed6d1beb4a44cd26"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.836458 639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:38:10.838796 654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.839956 639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:10.840080 639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "a226bba587854e2e8390e75b31cd4a85"
format_stamp: "Formatted at 2026-09-25 16:38:10 on dist-test-slave-kxl8"
server_key: "e182744e5fa500e702dc7ac73c57ce11"
server_key_iv: "ca0acf1a8bb265c6ed6d1beb4a44cd26"
server_key_version: "encryptionkey@0"
I20260925 16:38:10.840193 639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:10.857627 639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:10.860555 639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:10.860759 639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:10.861411 639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:10.862219 639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:10.862284 639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.862354 639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:10.862392 639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:10.871896 639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38503
I20260925 16:38:10.871915 767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38503 every 8 connection(s)
I20260925 16:38:10.872787 639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:10.875172 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 639
I20260925 16:38:10.875295 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:10.875604 391 external_mini_cluster.cc:1428] Setting key cba85e64758f2acd28f650ed167de43b
Sep 25 16:38:10 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:10.886929 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:10.874538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:53095 (local address 127.0.97.254:41251)
0925 16:38:10.874678 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:10.874682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:10.875519 (+ 837us) server_negotiation.cc:407] Connection header received
0925 16:38:10.877242 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:10.877246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:10.877355 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:10.877448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:10.878557 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.879412 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.880514 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.880750 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.880859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:10.883537 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:10.883541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:10.883544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:10.883573 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:10.885006 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.885563 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.885567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.885569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.885683 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:10.885997 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:10.885999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:10.886000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:10.886163 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:10.886234 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:10.886705 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 16:38:10.886803 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:38:10.887910 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:10.874915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41251 (local address 127.0.97.194:53095)
0925 16:38:10.875362 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:10.875412 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:10.877037 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:10.877600 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:10.877606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:10.877947 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:10.878422 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:10.878436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.879526 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.879528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:10.880387 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:10.880396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:10.880896 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:10.880900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:10.881065 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:10.881739 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:10.881755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:10.883372 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:10.885118 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:10.885123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:10.885139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:10.885467 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:10.885778 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:10.885780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:10.885782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:10.885926 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:10.886245 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:10.886250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:10.886478 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 16:38:10.886640 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 16:38:10.886900 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260925 16:38:10.890116 768 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41251
I20260925 16:38:10.890375 768 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:10.890877 768 heartbeater.cc:511] Master 127.0.97.254:41251 requested a full tablet report, sending...
I20260925 16:38:10.892078 443 ts_manager.cc:194] Registered new tserver with Master: a226bba587854e2e8390e75b31cd4a85 (127.0.97.194:38503)
I20260925 16:38:10.892974 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:53095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:10.904881 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:11.018460 773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:11.018864 773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:11.018944 773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:11.023105 773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:11.023185 773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:11.023275 773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:11.029363 773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:11.030864 773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:11.031883 773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:11.041246 779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:11.041271 778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:11.041354 773 server_base.cc:1034] running on GCE node
W20260925 16:38:11.041265 781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:11.042022 773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:11.042634 773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:11.043838 773 hybrid_clock.cc:648] HybridClock initialized: now 1790354291043822 us; error 28 us; skew 500 ppm
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:11.046641 773 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:11.047479 773 webserver.cc:460] Webserver started at http://127.0.97.195:36937/ using document root <none> and password file <none>
I20260925 16:38:11.048024 773 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:11.048110 773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:11.048331 773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:11.050417 773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "20d4932e05b645fdbb50111c7418f07e"
format_stamp: "Formatted at 2026-09-25 16:38:11 on dist-test-slave-kxl8"
server_key: "dd2ce7637f41e316ed0b8f65217d3c51"
server_key_iv: "81192f141ff98682156e823e39dd7618"
server_key_version: "encryptionkey@0"
I20260925 16:38:11.050987 773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "20d4932e05b645fdbb50111c7418f07e"
format_stamp: "Formatted at 2026-09-25 16:38:11 on dist-test-slave-kxl8"
server_key: "dd2ce7637f41e316ed0b8f65217d3c51"
server_key_iv: "81192f141ff98682156e823e39dd7618"
server_key_version: "encryptionkey@0"
I20260925 16:38:11.054628 773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:11.056965 788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:11.058223 773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:11.058362 773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "20d4932e05b645fdbb50111c7418f07e"
format_stamp: "Formatted at 2026-09-25 16:38:11 on dist-test-slave-kxl8"
server_key: "dd2ce7637f41e316ed0b8f65217d3c51"
server_key_iv: "81192f141ff98682156e823e39dd7618"
server_key_version: "encryptionkey@0"
I20260925 16:38:11.058467 773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:11.073768 773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:11.077028 773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:11.077268 773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:11.077898 773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:11.078758 773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:11.078826 773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:11.078903 773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:11.078943 773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:11.088418 773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:33743
I20260925 16:38:11.088440 901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:33743 every 8 connection(s)
I20260925 16:38:11.089361 773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:11.090930 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 773
I20260925 16:38:11.091044 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:11.091336 391 external_mini_cluster.cc:1428] Setting key f706cd49556bc93cc721a54f0b57167b
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:11.103730 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:11.091363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:44061 (local address 127.0.97.254:41251)
0925 16:38:11.091511 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:11.091516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:11.092513 (+ 997us) server_negotiation.cc:407] Connection header received
0925 16:38:11.094314 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:11.094317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:11.094395 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:11.094477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:11.095661 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.096189 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.097319 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.097545 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.097655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:11.100102 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:11.100106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:11.100109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:11.100138 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:11.101696 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.102174 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.102179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.102181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.102265 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.102649 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.102652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.102654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.102833 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:11.102940 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:11.103412 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 16:38:11.103540 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:38:11.104480 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:11.091811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41251 (local address 127.0.97.195:44061)
0925 16:38:11.092362 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:11.092421 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:11.094025 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:11.094673 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:11.094680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:11.095032 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:11.095529 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:11.095542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.096301 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.096305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:11.097179 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:11.097188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.097688 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.097692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:11.097856 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:11.098411 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:11.098426 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:11.099947 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:11.101801 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:11.101807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:11.101823 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:11.102067 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:11.102357 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:11.102360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:11.102362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:11.102549 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:11.102930 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:11.102941 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:11.103186 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 16:38:11.103370 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 16:38:11.103698 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":156,"threads_started":1}
I20260925 16:38:11.105624 902 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41251
I20260925 16:38:11.105865 902 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:11.106385 902 heartbeater.cc:511] Master 127.0.97.254:41251 requested a full tablet report, sending...
I20260925 16:38:11.107280 442 ts_manager.cc:194] Registered new tserver with Master: 20d4932e05b645fdbb50111c7418f07e (127.0.97.195:33743)
I20260925 16:38:11.107899 442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:44061
I20260925 16:38:11.117774 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:11.143525 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:11.133880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.97.193:34373)
0925 16:38:11.134192 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:11.134205 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:11.134228 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:38:11.134309 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:11.134314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:11.134506 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:11.134646 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:11.135256 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.136068 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.136969 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.137425 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.137511 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:11.139906 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:11.139914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:11.139927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:11.139951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:11.141720 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.142148 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.142153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.142154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.142236 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.142486 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.142488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.142489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.142736 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:11.142912 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:11.143077 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:38:11.143258 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
W20260925 16:38:11.144567 548 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57554
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:11.170976 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:11.161907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.193:34373)
0925 16:38:11.162078 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:11.162082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:11.162150 (+ 68us) server_negotiation.cc:407] Connection header received
0925 16:38:11.162292 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:11.162295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:11.162349 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:11.162466 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:11.163160 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.163875 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.164865 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:11.165135 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:11.165242 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:11.167957 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:11.167960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:11.167963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:11.167985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:11.169318 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.169730 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.169735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.169737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.169821 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:11.170100 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:11.170104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:11.170105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:11.170277 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:11.170408 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:11.170687 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 16:38:11.170813 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:38:11.171973 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 505
I20260925 16:38:11.178277 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 639
I20260925 16:38:11.186002 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 773
I20260925 16:38:11.191764 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 411
2026-09-25T16:38:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2215 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[916](info): set up 2 sockets
Sep 25 16:38:11 dist-test-slave-kxl8 krb5kdc[916](info): commencing operation
krb5kdc: starting...
W20260925 16:38:12.304586 391 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 16:38:12.434019 391 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.129s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:12 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:12.680398 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44565
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:12.792039 929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:12.792362 929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:12.792445 929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:12.796618 929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:12.796716 929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:12.796754 929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:12.796790 929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:12.796828 929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:12.802127 929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:12.803241 929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:12.804282 929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:12.812821 935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:12.812904 929 server_base.cc:1034] running on GCE node
W20260925 16:38:12.812844 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:12.812844 934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:12.813686 929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:12.814821 929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:12.816000 929 hybrid_clock.cc:648] HybridClock initialized: now 1790354292815989 us; error 25 us; skew 500 ppm
Sep 25 16:38:12 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:12.818676 929 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:12.819494 929 webserver.cc:460] Webserver started at http://127.0.97.254:39349/ using document root <none> and password file <none>
I20260925 16:38:12.820009 929 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:12.820065 929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:12.820230 929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:12.822213 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc"
format_stamp: "Formatted at 2026-09-25 16:38:12 on dist-test-slave-kxl8"
server_key: "61511d082f7acacaf0d2ff9d4e07281e"
server_key_iv: "b2d6b454fe3e01af2d758ca5d25ec063"
server_key_version: "encryptionkey@0"
I20260925 16:38:12.822693 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc"
format_stamp: "Formatted at 2026-09-25 16:38:12 on dist-test-slave-kxl8"
server_key: "61511d082f7acacaf0d2ff9d4e07281e"
server_key_iv: "b2d6b454fe3e01af2d758ca5d25ec063"
server_key_version: "encryptionkey@0"
I20260925 16:38:12.826225 929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:12.828579 944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:12.829851 929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:12.829972 929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc"
format_stamp: "Formatted at 2026-09-25 16:38:12 on dist-test-slave-kxl8"
server_key: "61511d082f7acacaf0d2ff9d4e07281e"
server_key_iv: "b2d6b454fe3e01af2d758ca5d25ec063"
server_key_version: "encryptionkey@0"
I20260925 16:38:12.830073 929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:12.843104 929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:12.846302 929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:12.846486 929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:12.853047 929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44565
I20260925 16:38:12.853107 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44565 every 8 connection(s)
I20260925 16:38:12.854028 929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:12.856257 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 929
I20260925 16:38:12.856345 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:12.856600 391 external_mini_cluster.cc:1428] Setting key 4b7b37220550e0e0daf8d5b7642d0234
I20260925 16:38:12.856912 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:12.863978 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc: Bootstrap starting.
Sep 25 16:38:12 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:12.866690 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:12.867646 997 log.cc:824] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:12.869880 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc: No bootstrap required, opened a new log
I20260925 16:38:12.871294 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:12.857963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.97.254:44565)
0925 16:38:12.858359 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:12.858379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:12.858418 (+ 39us) server_negotiation.cc:407] Connection header received
0925 16:38:12.859232 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:12.859255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:12.859663 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:12.860028 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:12.860620 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:12.862126 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:12.862941 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:12.863604 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:12.863777 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:12.866367 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:12.866373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:12.866388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:12.866416 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:12.868273 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:12.868691 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:12.868697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:12.868702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:12.868836 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:12.869076 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:12.869079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:12.869081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:12.869573 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:12.869777 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:12.869958 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 16:38:12.870405 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260925 16:38:12.873708 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } }
I20260925 16:38:12.873961 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:12.874032 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da1009d8c7e48fbb172f99a6e6a3dbc, State: Initialized, Role: FOLLOWER
I20260925 16:38:12.874418 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } }
I20260925 16:38:12.874554 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:12.874612 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:12.874703 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:12.875589 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } }
I20260925 16:38:12.875909 997 leader_election.cc:302] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da1009d8c7e48fbb172f99a6e6a3dbc; no voters:
I20260925 16:38:12.876200 997 leader_election.cc:288] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:12.876333 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:12.876551 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [term 1 LEADER]: Becoming Leader. State: Replica: 7da1009d8c7e48fbb172f99a6e6a3dbc, State: Running, Role: LEADER
I20260925 16:38:12.876838 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } }
I20260925 16:38:12.877415 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:12.878402 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da1009d8c7e48fbb172f99a6e6a3dbc. Latest consensus state: current_term: 1 leader_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } } }
I20260925 16:38:12.878531 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:12.878818 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da1009d8c7e48fbb172f99a6e6a3dbc" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44565 } } }
I20260925 16:38:12.878918 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:12.879252 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:12.882180 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:12.887704 1011 catalog_manager.cc:1347] Generated new cluster ID: 0f413a5c6be244558e64d4ec9abeb1b3
I20260925 16:38:12.887773 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:12.899330 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:12.900236 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:12.906060 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da1009d8c7e48fbb172f99a6e6a3dbc: Generated new TSK 0
I20260925 16:38:12.906697 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:12.940004 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:13.050262 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:13.050500 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:13.050572 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:13.054636 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:13.054692 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:13.054754 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:13.059942 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:13.060922 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:13.061919 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:13.071184 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.071255 1023 server_base.cc:1034] running on GCE node
W20260925 16:38:13.071110 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:13.071120 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.071743 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:13.072335 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:13.073503 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790354293073488 us; error 26 us; skew 500 ppm
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:13.076208 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:13.077067 1023 webserver.cc:460] Webserver started at http://127.0.97.193:44763/ using document root <none> and password file <none>
I20260925 16:38:13.077651 1023 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:13.077719 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:13.077935 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:13.079990 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "15053149756748b6bfc27e407cccc9cf"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "79cd29aea6393b6717c41caf671f3f40"
server_key_iv: "d6b58c7fa37382aa0bbc4df37ecc1ce0"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.080482 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "15053149756748b6bfc27e407cccc9cf"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "79cd29aea6393b6717c41caf671f3f40"
server_key_iv: "d6b58c7fa37382aa0bbc4df37ecc1ce0"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.083994 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:38:13.086386 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.087533 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:13.087625 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "15053149756748b6bfc27e407cccc9cf"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "79cd29aea6393b6717c41caf671f3f40"
server_key_iv: "d6b58c7fa37382aa0bbc4df37ecc1ce0"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.087719 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:13.115217 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:13.118412 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:13.118561 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:13.119092 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:13.119845 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:13.119891 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.119930 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:13.119952 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.128255 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:35919
I20260925 16:38:13.128274 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:35919 every 8 connection(s)
I20260925 16:38:13.129128 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:13.136495 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1023
I20260925 16:38:13.136582 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:13.136824 391 external_mini_cluster.cc:1428] Setting key 53e703848c13114d3dee36854d35156a
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:13.142861 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.130815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:56055 (local address 127.0.97.254:44565)
0925 16:38:13.130944 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.130947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.131750 (+ 803us) server_negotiation.cc:407] Connection header received
0925 16:38:13.133062 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.133065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.133149 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.133234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.134396 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.134948 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.136046 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.136288 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.136400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.139071 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.139080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.139082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.139104 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.140850 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.141382 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.141385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.141387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.141473 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.141802 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.141805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.141807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.141985 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.142081 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.142623 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.142746 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:13.144068 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.131196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44565 (local address 127.0.97.193:56055)
0925 16:38:13.131603 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.131643 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.132854 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.133443 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.133450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.133793 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:13.134251 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.134264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.135065 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.135069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.135925 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.135934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.136437 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.136441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.136653 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.137266 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:13.137338 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:13.138918 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:13.140962 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.140968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.140982 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.141233 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.141564 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.141567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.141569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.141699 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.142087 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:13.142095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:13.142346 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.142554 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.142937 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
I20260925 16:38:13.145303 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44565
I20260925 16:38:13.145556 1152 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:13.146129 1152 heartbeater.cc:511] Master 127.0.97.254:44565 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:13.147631 961 ts_manager.cc:194] Registered new tserver with Master: 15053149756748b6bfc27e407cccc9cf (127.0.97.193:35919)
I20260925 16:38:13.148671 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:56055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:13.163775 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:13.274323 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:13.274598 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:13.274643 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:13.278816 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:13.278877 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:13.278954 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:13.284243 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:13.285348 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:13.286358 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:13.295884 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.296010 1157 server_base.cc:1034] running on GCE node
W20260925 16:38:13.295871 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:13.295900 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.296566 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:13.297204 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:13.298362 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790354293298344 us; error 29 us; skew 500 ppm
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:13.301178 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:13.302049 1157 webserver.cc:460] Webserver started at http://127.0.97.194:45299/ using document root <none> and password file <none>
I20260925 16:38:13.302603 1157 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:13.302663 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:13.302891 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:13.304968 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "8bce40eacfb643049585cd273e0ca4a1"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "27bd511133ec3ea55e775ebd3b5861e7"
server_key_iv: "9e864bddb7b50f479d62b68106b1204f"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.305548 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "8bce40eacfb643049585cd273e0ca4a1"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "27bd511133ec3ea55e775ebd3b5861e7"
server_key_iv: "9e864bddb7b50f479d62b68106b1204f"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.309180 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:38:13.311569 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.312778 1157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:13.312906 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "8bce40eacfb643049585cd273e0ca4a1"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "27bd511133ec3ea55e775ebd3b5861e7"
server_key_iv: "9e864bddb7b50f479d62b68106b1204f"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.313014 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:13.335485 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:13.338863 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:13.339066 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:13.339709 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:13.340528 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:13.340595 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.340665 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:13.340706 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.349244 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38709
I20260925 16:38:13.349306 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38709 every 8 connection(s)
I20260925 16:38:13.350168 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:13.359735 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1157
I20260925 16:38:13.359859 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:13.360123 391 external_mini_cluster.cc:1428] Setting key 0d977b3b19c6148f745d749711724bcd
I20260925 16:38:13.363976 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.351963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:57427 (local address 127.0.97.254:44565)
0925 16:38:13.352096 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.352099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.352936 (+ 837us) server_negotiation.cc:407] Connection header received
0925 16:38:13.354333 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.354336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.354392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.354495 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.355628 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.356155 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.357271 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.357523 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.357631 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.360310 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.360314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.360317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.360347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.362008 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.362501 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.362506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.362508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.362586 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.362935 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.362939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.362941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.363120 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.363219 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.363698 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.363817 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:13.364825 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.352345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44565 (local address 127.0.97.194:57427)
0925 16:38:13.352792 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.352833 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.354122 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.354671 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.354678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.355021 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:13.355487 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.355500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.356284 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.356293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.357152 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.357161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.357674 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.357678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.357836 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.358424 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:13.358439 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:13.360066 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:13.362135 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.362140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.362154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.362401 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.362704 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.362707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.362709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.362845 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.363228 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:13.363237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:13.363472 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.363659 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.363965 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
I20260925 16:38:13.365921 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44565
I20260925 16:38:13.366158 1286 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:13.366627 1286 heartbeater.cc:511] Master 127.0.97.254:44565 requested a full tablet report, sending...
I20260925 16:38:13.367676 961 ts_manager.cc:194] Registered new tserver with Master: 8bce40eacfb643049585cd273e0ca4a1 (127.0.97.194:38709)
I20260925 16:38:13.368505 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:57427
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:13.388818 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:13.499620 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:13.499871 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:13.499914 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:13.504074 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:13.504129 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:13.504215 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:13.509359 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:13.510411 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:13.511416 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:13.521073 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.521198 1291 server_base.cc:1034] running on GCE node
W20260925 16:38:13.521133 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:13.521072 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:13.521821 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:13.522432 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:13.523583 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790354293523568 us; error 31 us; skew 500 ppm
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:13.526422 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:13.527288 1291 webserver.cc:460] Webserver started at http://127.0.97.195:34947/ using document root <none> and password file <none>
I20260925 16:38:13.527830 1291 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:13.527889 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:13.528111 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:13.530172 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "92d5d4eb04de4b829eab62b7f12b8952"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "58b7d24933baaa084c17e569ebcf89ab"
server_key_iv: "5b13a8ec65b0c466d68d6ef6aec67aec"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.530736 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "92d5d4eb04de4b829eab62b7f12b8952"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "58b7d24933baaa084c17e569ebcf89ab"
server_key_iv: "5b13a8ec65b0c466d68d6ef6aec67aec"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.534394 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 16:38:13.536801 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.538094 1291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:13.538221 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "92d5d4eb04de4b829eab62b7f12b8952"
format_stamp: "Formatted at 2026-09-25 16:38:13 on dist-test-slave-kxl8"
server_key: "58b7d24933baaa084c17e569ebcf89ab"
server_key_iv: "5b13a8ec65b0c466d68d6ef6aec67aec"
server_key_version: "encryptionkey@0"
I20260925 16:38:13.538321 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:13.552006 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:13.555335 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:13.555529 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:13.556120 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:13.556946 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:13.556998 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.557062 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:13.557098 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:13.566141 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:45891
I20260925 16:38:13.566179 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:45891 every 8 connection(s)
I20260925 16:38:13.567046 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:13.575243 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1291
I20260925 16:38:13.575380 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:13.575650 391 external_mini_cluster.cc:1428] Setting key 729df86319908022663dcf43c1e5a381
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:13.580562 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.568852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:58025 (local address 127.0.97.254:44565)
0925 16:38:13.568991 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.568995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.569795 (+ 800us) server_negotiation.cc:407] Connection header received
0925 16:38:13.571125 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.571128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.571189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.571289 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.572418 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.572948 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.574125 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.574351 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.574456 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.576952 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.576961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.576963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.576986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.578499 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.579056 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.579061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.579066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.579148 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.579479 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.579482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.579484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.579668 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.579774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.580257 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.580387 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:13.581368 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.569171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44565 (local address 127.0.97.195:58025)
0925 16:38:13.569631 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.569676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.570900 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.571451 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.571458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.571810 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:13.572278 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.572290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.573084 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.573086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.573997 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.574006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.574492 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.574496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.574658 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.575260 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:13.575275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:13.576799 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:13.578688 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.578694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.578709 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.578958 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.579240 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:13.579243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:13.579245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:13.579382 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:13.579782 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:13.579790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:13.580024 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.580207 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.580514 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260925 16:38:13.582397 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44565
I20260925 16:38:13.582598 1420 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:13.583009 1420 heartbeater.cc:511] Master 127.0.97.254:44565 requested a full tablet report, sending...
I20260925 16:38:13.583902 961 ts_manager.cc:194] Registered new tserver with Master: 92d5d4eb04de4b829eab62b7f12b8952 (127.0.97.195:45891)
I20260925 16:38:13.584439 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:58025
I20260925 16:38:13.589313 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:38:13.600232 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.592803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.0.97.254:44565)
0925 16:38:13.592925 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.592929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.593170 (+ 241us) server_negotiation.cc:407] Connection header received
0925 16:38:13.593319 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.593321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.593368 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.593454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.594218 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.594723 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.595516 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.595738 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.595838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.597165 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.597169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.597172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.597198 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.598623 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.599022 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.599027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.599029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.599106 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.599377 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.599382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.599384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.599593 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.599724 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.599956 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.600065 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:38:13.604827 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:13.607770 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:13.621555 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.615485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38709 (local address 127.0.0.1:56450)
0925 16:38:13.615795 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.615839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.616008 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.616752 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.616758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.616784 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:13.617261 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.617273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.618584 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.618587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.619932 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.619943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.620981 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.620984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.621086 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621127 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.621253 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.621397 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":46,"thread_start_us":92,"threads_started":1}
I20260925 16:38:13.621742 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.616603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.195:45891)
0925 16:38:13.616994 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.617005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.617027 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:38:13.617101 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.617108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.617310 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.617449 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:13.617858 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.618833 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.620311 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.621075 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.621175 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621324 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.621486 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.621602 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":93,"threads_started":1}
I20260925 16:38:13.621989 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.615569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45891 (local address 127.0.0.1:48564)
0925 16:38:13.616429 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.616443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.616533 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.617625 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.617628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.617644 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:13.617738 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.617745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.618948 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.618953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.620187 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.620197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.621201 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.621204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.621344 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621366 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.621458 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.621529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260925 16:38:13.622217 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.616359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.0.97.193:35919)
0925 16:38:13.617085 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.617102 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.617513 (+ 411us) server_negotiation.cc:407] Connection header received
0925 16:38:13.617592 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.617599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.617765 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.617899 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:13.618223 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.619223 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.620547 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.621184 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.621277 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621422 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.621959 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.622060 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":91,"threads_started":1}
I20260925 16:38:13.622339 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.615587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.0.97.194:38709)
0925 16:38:13.616260 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.616274 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.616301 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:38:13.616375 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.616382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.616584 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.616738 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:13.617384 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.618444 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.620075 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.620858 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.620968 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621128 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.621630 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.622201 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":82,"threads_started":1}
I20260925 16:38:13.622507 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.616206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:35919 (local address 127.0.0.1:40292)
0925 16:38:13.617316 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.617330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.617426 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.618022 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.618024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.618037 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:13.618108 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.618112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.619358 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.619362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.620437 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.620443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.621479 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.621482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.621558 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.621593 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.621680 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.622372 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260925 16:38:13.624675 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 90a9c92bc8c349019ed8768c1c0de569 (DEFAULT_TABLE table=test-table [id=c2975216875d4e2885a795e54a29c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:13.624672 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 90a9c92bc8c349019ed8768c1c0de569 (DEFAULT_TABLE table=test-table [id=c2975216875d4e2885a795e54a29c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:13.625625 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a9c92bc8c349019ed8768c1c0de569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:13.625646 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a9c92bc8c349019ed8768c1c0de569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:13.626927 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 90a9c92bc8c349019ed8768c1c0de569 (DEFAULT_TABLE table=test-table [id=c2975216875d4e2885a795e54a29c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:13.627821 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a9c92bc8c349019ed8768c1c0de569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:13.631425 1437 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Bootstrap starting.
I20260925 16:38:13.632989 1438 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Bootstrap starting.
I20260925 16:38:13.633668 1437 tablet_bootstrap.cc:654] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:13.634825 1439 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Bootstrap starting.
I20260925 16:38:13.634876 1437 log.cc:824] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:13.635634 1438 tablet_bootstrap.cc:654] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:13.636279 1438 log.cc:824] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:13.636718 1439 tablet_bootstrap.cc:654] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:13.637063 1437 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: No bootstrap required, opened a new log
I20260925 16:38:13.637303 1437 ts_tablet_manager.cc:1397] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:38:13.637943 1438 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: No bootstrap required, opened a new log
I20260925 16:38:13.638118 1438 ts_tablet_manager.cc:1397] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:13.638216 1439 log.cc:824] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:13.640342 1437 raft_consensus.cc:348] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.640553 1437 raft_consensus.cc:374] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:13.640586 1437 raft_consensus.cc:729] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d5d4eb04de4b829eab62b7f12b8952, State: Initialized, Role: FOLLOWER
I20260925 16:38:13.641074 1437 consensus_queue.cc:260] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.641809 1420 heartbeater.cc:503] Master 127.0.97.254:44565 was elected leader, sending a full tablet report...
I20260925 16:38:13.642127 1437 ts_tablet_manager.cc:1428] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:13.642308 1439 tablet_bootstrap.cc:492] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: No bootstrap required, opened a new log
I20260925 16:38:13.642504 1439 ts_tablet_manager.cc:1397] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 16:38:13.641620 1438 raft_consensus.cc:348] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.643000 1438 raft_consensus.cc:374] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:13.643074 1438 raft_consensus.cc:729] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bce40eacfb643049585cd273e0ca4a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:13.643481 1438 consensus_queue.cc:260] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.644114 1286 heartbeater.cc:503] Master 127.0.97.254:44565 was elected leader, sending a full tablet report...
I20260925 16:38:13.644595 1438 ts_tablet_manager.cc:1428] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:38:13.646019 1439 raft_consensus.cc:348] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.646297 1439 raft_consensus.cc:374] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:13.646361 1439 raft_consensus.cc:729] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15053149756748b6bfc27e407cccc9cf, State: Initialized, Role: FOLLOWER
I20260925 16:38:13.646776 1439 consensus_queue.cc:260] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.647308 1152 heartbeater.cc:503] Master 127.0.97.254:44565 was elected leader, sending a full tablet report...
I20260925 16:38:13.647648 1439 ts_tablet_manager.cc:1428] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 16:38:13.817979 1421 tablet.cc:2369] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:38:13.851249 1287 tablet.cc:2369] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:13.867437 1445 raft_consensus.cc:482] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:13.867638 1445 raft_consensus.cc:504] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.868856 1445 leader_election.cc:288] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bce40eacfb643049585cd273e0ca4a1 (127.0.97.194:38709), 92d5d4eb04de4b829eab62b7f12b8952 (127.0.97.195:45891)
I20260925 16:38:13.873899 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.869461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:51031 (local address 127.0.97.194:38709)
0925 16:38:13.869590 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.869596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.869756 (+ 160us) server_negotiation.cc:407] Connection header received
0925 16:38:13.869919 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.869924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.869977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.870055 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:13.870581 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.871426 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.872687 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.873332 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.873447 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.873499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.873702 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.873775 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:38:13.874084 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.869083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45891 (local address 127.0.97.193:50107)
0925 16:38:13.870060 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.870074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.870230 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.870479 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.870482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.870496 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:13.870616 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.870623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.871589 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.871592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.872789 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.872799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.873643 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.873646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.873753 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.873779 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.873872 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.873939 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":18,"thread_start_us":85,"threads_started":1}
I20260925 16:38:13.874066 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.869244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:50107 (local address 127.0.97.195:45891)
0925 16:38:13.869388 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.869392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.870160 (+ 768us) server_negotiation.cc:407] Connection header received
0925 16:38:13.870342 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.870345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.870393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.870470 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:13.870729 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.871464 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.872895 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.873517 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.873643 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.873692 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.873890 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.873960 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:38:13.874219 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.869083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38709 (local address 127.0.97.193:51031)
0925 16:38:13.869666 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:13.869679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:13.869810 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:13.870088 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:13.870091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:13.870115 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:13.870242 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.870247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.871558 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.871561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.872576 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:13.872585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.873441 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.873445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:13.873548 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.873573 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:13.874020 (+ 447us) client_negotiation.cc:240] Negotiation successful
0925 16:38:13.874092 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":36,"thread_start_us":89,"threads_started":1}
I20260925 16:38:13.874930 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a9c92bc8c349019ed8768c1c0de569" candidate_uuid: "15053149756748b6bfc27e407cccc9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d5d4eb04de4b829eab62b7f12b8952" is_pre_election: true
I20260925 16:38:13.875087 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a9c92bc8c349019ed8768c1c0de569" candidate_uuid: "15053149756748b6bfc27e407cccc9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bce40eacfb643049585cd273e0ca4a1" is_pre_election: true
I20260925 16:38:13.875233 1374 raft_consensus.cc:2454] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15053149756748b6bfc27e407cccc9cf in term 0.
I20260925 16:38:13.875320 1240 raft_consensus.cc:2454] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15053149756748b6bfc27e407cccc9cf in term 0.
I20260925 16:38:13.875723 1039 leader_election.cc:302] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15053149756748b6bfc27e407cccc9cf, 8bce40eacfb643049585cd273e0ca4a1; no voters:
I20260925 16:38:13.876127 1445 raft_consensus.cc:2781] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:13.876214 1445 raft_consensus.cc:482] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:13.876253 1445 raft_consensus.cc:3037] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:13.877377 1445 raft_consensus.cc:504] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.877656 1445 leader_election.cc:288] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [CANDIDATE]: Term 1 election: Requested vote from peers 8bce40eacfb643049585cd273e0ca4a1 (127.0.97.194:38709), 92d5d4eb04de4b829eab62b7f12b8952 (127.0.97.195:45891)
I20260925 16:38:13.877998 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a9c92bc8c349019ed8768c1c0de569" candidate_uuid: "15053149756748b6bfc27e407cccc9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d5d4eb04de4b829eab62b7f12b8952"
I20260925 16:38:13.878021 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a9c92bc8c349019ed8768c1c0de569" candidate_uuid: "15053149756748b6bfc27e407cccc9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bce40eacfb643049585cd273e0ca4a1"
I20260925 16:38:13.878119 1374 raft_consensus.cc:3037] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:13.878118 1240 raft_consensus.cc:3037] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:13.879371 1240 raft_consensus.cc:2454] T 90a9c92bc8c349019ed8768c1c0de569 P 8bce40eacfb643049585cd273e0ca4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15053149756748b6bfc27e407cccc9cf in term 1.
I20260925 16:38:13.879371 1374 raft_consensus.cc:2454] T 90a9c92bc8c349019ed8768c1c0de569 P 92d5d4eb04de4b829eab62b7f12b8952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15053149756748b6bfc27e407cccc9cf in term 1.
I20260925 16:38:13.879729 1039 leader_election.cc:302] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15053149756748b6bfc27e407cccc9cf, 8bce40eacfb643049585cd273e0ca4a1; no voters:
I20260925 16:38:13.879930 1445 raft_consensus.cc:2781] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:13.880126 1445 raft_consensus.cc:686] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [term 1 LEADER]: Becoming Leader. State: Replica: 15053149756748b6bfc27e407cccc9cf, State: Running, Role: LEADER
W20260925 16:38:13.880376 1153 tablet.cc:2369] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:13.880451 1445 consensus_queue.cc:237] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } }
I20260925 16:38:13.883522 960 catalog_manager.cc:5495] T 90a9c92bc8c349019ed8768c1c0de569 P 15053149756748b6bfc27e407cccc9cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 15053149756748b6bfc27e407cccc9cf (127.0.97.193). New cstate: current_term: 1 leader_uuid: "15053149756748b6bfc27e407cccc9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bce40eacfb643049585cd273e0ca4a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d5d4eb04de4b829eab62b7f12b8952" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15053149756748b6bfc27e407cccc9cf" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35919 } health_report { overall_health: HEALTHY } } }
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:13.923713 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.914465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.97.193:35919)
0925 16:38:13.914674 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.914682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.914761 (+ 79us) server_negotiation.cc:407] Connection header received
0925 16:38:13.914864 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.914867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.914913 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.914989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.915717 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.916312 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.917395 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.917639 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.917761 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.920393 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.920412 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.920427 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.920451 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.921962 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.922397 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.922403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.922404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.922480 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.922784 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.922788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.922790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.923030 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.923293 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.923423 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.923545 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:13.950253 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.941827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.193:35919)
0925 16:38:13.941982 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.941987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.942139 (+ 152us) server_negotiation.cc:407] Connection header received
0925 16:38:13.942227 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.942230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.942289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.942374 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.943077 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.943640 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.944501 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.944715 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.944822 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.947256 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.947260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.947262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.947287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.948678 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.949065 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.949068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.949069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.949146 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.949442 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.949445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.949446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.949616 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.949722 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.949952 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.950042 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:38:13 dist-test-slave-kxl8 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:13.975337 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:13.967092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.97.193:35919)
0925 16:38:13.967258 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:13.967263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:13.967381 (+ 118us) server_negotiation.cc:407] Connection header received
0925 16:38:13.967556 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:13.967559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:13.967609 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:13.967699 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:13.968355 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.968902 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.969827 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:13.970049 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:13.970165 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:13.972488 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:13.972492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:13.972494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:13.972518 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:13.973748 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.974159 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.974162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.974164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.974235 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:13.974512 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:13.974516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:13.974518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:13.974684 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:13.974839 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:13.975047 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 16:38:13.975176 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 16:38:13.976651 1066 tablet_service.cc:2908] Rejecting scan request for tablet 90a9c92bc8c349019ed8768c1c0de569: Uninitialized: clean time has not yet been initialized
I20260925 16:38:13.978812 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1023
I20260925 16:38:13.985971 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1157
I20260925 16:38:13.991689 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1291
I20260925 16:38:13.997152 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 929
2026-09-25T16:38:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2805 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:14 dist-test-slave-kxl8 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:14 dist-test-slave-kxl8 krb5kdc[1462](info): set up 2 sockets
Sep 25 16:38:14 dist-test-slave-kxl8 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260925 16:38:15.057003 391 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:15.180672 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44253
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:15.290126 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:15.290436 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:15.290504 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:15.295176 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:15.295256 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:38:15.295286 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:15.295332 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:15.295369 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:15.295405 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:15.300649 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:15.301790 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:15.302802 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:38:15.311339 1475 server_base.cc:1034] running on GCE node
W20260925 16:38:15.311321 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.311285 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.311280 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:15.312024 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:15.313175 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:15.314374 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790354295314344 us; error 46 us; skew 500 ppm
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:15.317121 1475 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:15.318037 1475 webserver.cc:460] Webserver started at http://127.0.97.254:40745/ using document root <none> and password file <none>
I20260925 16:38:15.318599 1475 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:15.318684 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:15.318907 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:15.320937 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "5cc817064bf34ffab96d07dd92ad0294"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "9d4b8f3fcd8b5fe362b84e20ddb74be3"
server_key_iv: "8c9aa063cfead55e0524cbcc3e077aa3"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.321511 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "5cc817064bf34ffab96d07dd92ad0294"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "9d4b8f3fcd8b5fe362b84e20ddb74be3"
server_key_iv: "8c9aa063cfead55e0524cbcc3e077aa3"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.325070 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:38:15.327481 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:15.328678 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:15.328806 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "5cc817064bf34ffab96d07dd92ad0294"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "9d4b8f3fcd8b5fe362b84e20ddb74be3"
server_key_iv: "8c9aa063cfead55e0524cbcc3e077aa3"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.328909 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:15.348543 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:15.351958 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:15.352160 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:15.359936 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44253
I20260925 16:38:15.360008 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44253 every 8 connection(s)
I20260925 16:38:15.360888 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:15.363633 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:15.366626 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1475
I20260925 16:38:15.366775 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:15.367027 391 external_mini_cluster.cc:1428] Setting key b761a515e7a175c94892640af79d61c9
I20260925 16:38:15.369535 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Bootstrap starting.
I20260925 16:38:15.371636 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:15.372350 1553 log.cc:824] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:15.374476 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: No bootstrap required, opened a new log
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:15.378294 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } }
I20260925 16:38:15.378582 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:15.378641 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc817064bf34ffab96d07dd92ad0294, State: Initialized, Role: FOLLOWER
I20260925 16:38:15.379132 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } }
I20260925 16:38:15.379312 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:15.379393 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:15.379532 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:15.380741 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } }
I20260925 16:38:15.381124 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc817064bf34ffab96d07dd92ad0294; no voters:
I20260925 16:38:15.381498 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:15.381606 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:15.381850 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [term 1 LEADER]: Becoming Leader. State: Replica: 5cc817064bf34ffab96d07dd92ad0294, State: Running, Role: LEADER
I20260925 16:38:15.381889 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.368356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.0.97.254:44253)
0925 16:38:15.368823 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:15.368838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:15.368871 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:38:15.369550 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:15.369569 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:15.369939 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:15.370255 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:15.371007 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.372038 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.373004 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.373697 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.373883 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.376539 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:15.376544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:15.376557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:15.376583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:15.378358 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.378889 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.378897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.378903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.379043 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.379372 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.379376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.379377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.379983 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:15.380178 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:15.380450 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 16:38:15.380782 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":131,"threads_started":1}
I20260925 16:38:15.382182 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } }
I20260925 16:38:15.382745 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:15.383225 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc817064bf34ffab96d07dd92ad0294. Latest consensus state: current_term: 1 leader_uuid: "5cc817064bf34ffab96d07dd92ad0294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } } }
I20260925 16:38:15.383342 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:15.383790 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc817064bf34ffab96d07dd92ad0294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc817064bf34ffab96d07dd92ad0294" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44253 } } }
I20260925 16:38:15.383898 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294 [sys.catalog]: This master's current role is: LEADER
W20260925 16:38:15.387279 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:38:15.387344 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:38:15.387420 1563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:15.388211 1563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:15.393913 1563 catalog_manager.cc:1347] Generated new cluster ID: 553a14cf3f44496da1846ee45d949e40
I20260925 16:38:15.393983 1563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:15.418243 1563 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:15.419281 1563 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:15.428089 1563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Generated new TSK 0
I20260925 16:38:15.428807 1563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:15.460390 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44253
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:15.569646 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:15.569901 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:15.569970 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:15.573892 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:15.573952 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:15.574023 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:15.574100 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:15.579281 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:15.580338 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:15.581414 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:15.591012 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.591012 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.591012 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:15.591300 1579 server_base.cc:1034] running on GCE node
I20260925 16:38:15.591714 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:15.592330 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:15.593523 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790354295593489 us; error 45 us; skew 500 ppm
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:15.596717 1579 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:15.597709 1579 webserver.cc:460] Webserver started at http://127.0.97.193:36619/ using document root <none> and password file <none>
I20260925 16:38:15.598225 1579 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:15.598284 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:15.598440 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:15.600400 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "bfee1a21264e460793fa2f43e98bddde"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "bd980bb8098d7feb87e9a92b6388a359"
server_key_iv: "e8d2c6916c327d489762371421d682a6"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.600911 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "bfee1a21264e460793fa2f43e98bddde"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "bd980bb8098d7feb87e9a92b6388a359"
server_key_iv: "e8d2c6916c327d489762371421d682a6"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.604640 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:38:15.607002 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:15.608233 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:38:15.608362 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "bfee1a21264e460793fa2f43e98bddde"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "bd980bb8098d7feb87e9a92b6388a359"
server_key_iv: "e8d2c6916c327d489762371421d682a6"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.608469 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:15.629968 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:15.633265 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:15.633507 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:15.634980 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:15.635051 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:15.635146 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:15.635185 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:15.645723 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:42047
I20260925 16:38:15.645752 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:42047 every 8 connection(s)
I20260925 16:38:15.646646 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:15.648721 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.635740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:37011 (local address 127.0.97.254:44253)
0925 16:38:15.635881 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:15.635886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:15.636777 (+ 891us) server_negotiation.cc:407] Connection header received
0925 16:38:15.637889 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:15.637893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:15.637957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:15.638037 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:15.639780 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.640689 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.642077 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.642325 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.642439 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.645019 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:15.645025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:15.645031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:15.645065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:15.646799 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.647401 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.647404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.647406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.647489 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.647803 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.647805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.647806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.647969 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:15.648042 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:15.648457 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 16:38:15.648569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:15.649605 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.636181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.193:37011)
0925 16:38:15.636621 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:15.636669 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:15.637635 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:15.638485 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:15.638493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:15.638907 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:15.639600 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.639620 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.640829 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.640835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.641954 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.641963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.642475 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.642480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.642660 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.643270 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:15.643285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:15.644852 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:15.646915 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.646921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.646935 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.647279 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.647588 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.647590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.647592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.647720 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.648051 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:15.648056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:15.648265 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 16:38:15.648417 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:38:15.648741 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260925 16:38:15.650876 1710 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44253
I20260925 16:38:15.651046 1710 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:15.651520 1710 heartbeater.cc:511] Master 127.0.97.254:44253 requested a full tablet report, sending...
I20260925 16:38:15.653036 1506 ts_manager.cc:194] Registered new tserver with Master: bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047)
I20260925 16:38:15.654191 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:37011
I20260925 16:38:15.656790 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1579
I20260925 16:38:15.656901 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:15.657168 391 external_mini_cluster.cc:1428] Setting key 97b2219223a755c1adc3830149a28973
I20260925 16:38:15.659819 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.652858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:47291 (local address 127.0.97.254:44253)
0925 16:38:15.652987 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:15.652996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:15.653226 (+ 230us) server_negotiation.cc:407] Connection header received
0925 16:38:15.653363 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:15.653366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:15.653413 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:15.653492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:15.653903 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.654459 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.655377 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.655599 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.655709 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.656774 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:15.656778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:15.656780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:15.656802 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:15.658281 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.658680 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.658685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.658687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.658767 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.659032 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.659036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.659038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.659205 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:15.659302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:15.659554 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 16:38:15.659665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:15.659828 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.652783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.193:47291)
0925 16:38:15.653131 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:15.653147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:15.653265 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:15.653505 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:15.653508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:15.653706 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:15.653795 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.653800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.654574 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.654577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.655263 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.655272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.655709 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.655713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.655805 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.656313 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:15.656330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:15.656640 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:15.658385 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.658390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.658392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.658585 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.658860 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.658863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.658865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.658946 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.659303 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:15.659307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:15.659399 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:38:15.659501 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:38:15.659639 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:15.683936 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44253
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:15.794857 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:15.795121 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:15.795169 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:15.799520 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:15.799589 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:15.799631 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:15.799680 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:15.804932 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:15.806047 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:15.807013 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:15.816721 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.816738 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:15.816771 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:15.816991 1719 server_base.cc:1034] running on GCE node
I20260925 16:38:15.817451 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:15.818110 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:15.819274 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790354295819252 us; error 32 us; skew 500 ppm
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:15.821995 1719 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:15.822902 1719 webserver.cc:460] Webserver started at http://127.0.97.194:46379/ using document root <none> and password file <none>
I20260925 16:38:15.823499 1719 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:15.823585 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:15.823812 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:15.825886 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "7fb2d43d006d4e22b57d5ab8667793bd"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "52df868ff7d3616d61ae20585ccd051c"
server_key_iv: "91740cc00f75b46d862018bc5967a8fb"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.826422 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "7fb2d43d006d4e22b57d5ab8667793bd"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "52df868ff7d3616d61ae20585ccd051c"
server_key_iv: "91740cc00f75b46d862018bc5967a8fb"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.830125 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:15.832442 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:15.833724 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:15.833838 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "7fb2d43d006d4e22b57d5ab8667793bd"
format_stamp: "Formatted at 2026-09-25 16:38:15 on dist-test-slave-kxl8"
server_key: "52df868ff7d3616d61ae20585ccd051c"
server_key_iv: "91740cc00f75b46d862018bc5967a8fb"
server_key_version: "encryptionkey@0"
I20260925 16:38:15.833963 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:15.859102 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:15.862416 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:15.862620 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:15.864066 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:15.864133 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:15.864223 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:15.864266 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:15 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:15.875128 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:37139
I20260925 16:38:15.875187 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:37139 every 8 connection(s)
I20260925 16:38:15.876237 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:15.878024 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37553 (local address 127.0.97.254:44253)
0925 16:38:15.864937 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:15.864943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:15.865847 (+ 904us) server_negotiation.cc:407] Connection header received
0925 16:38:15.867177 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:15.867183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:15.867251 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:15.867317 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:15.869141 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.869887 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.871033 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.871267 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.871379 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.874263 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:15.874272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:15.874274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:15.874298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:15.875998 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.876498 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.876503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.876508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.876602 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.876965 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.876968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.876970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.877186 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:15.877316 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:15.877814 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 16:38:15.877908 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:38:15.878752 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.865148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.194:37553)
0925 16:38:15.865686 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:15.865745 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:15.866914 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:15.867772 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:15.867782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:15.868235 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:15.868966 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.868988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.870012 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.870015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.870911 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.870920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.871433 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.871437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.871591 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.872150 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:15.872172 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:15.874101 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:15.876107 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.876111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.876122 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.876401 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.876716 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.876719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.876721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.876881 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.877368 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:15.877375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:15.877602 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 16:38:15.877761 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 16:38:15.877990 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":167,"threads_started":1}
I20260925 16:38:15.879892 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1719
I20260925 16:38:15.880009 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:15.879994 1850 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44253
I20260925 16:38:15.880111 1850 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:15.880313 391 external_mini_cluster.cc:1428] Setting key 78f5aca5ddf94b474b840a7276e72f36
I20260925 16:38:15.880651 1850 heartbeater.cc:511] Master 127.0.97.254:44253 requested a full tablet report, sending...
I20260925 16:38:15.881626 1506 ts_manager.cc:194] Registered new tserver with Master: 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139)
I20260925 16:38:15.882365 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:37553
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:15.891330 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.882134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.194:52195)
0925 16:38:15.882427 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:15.882445 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:15.882534 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:15.883930 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:15.883936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:15.884378 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:15.884524 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.884578 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.885469 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.885475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.886502 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:15.886515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.886950 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.886957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:15.887085 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.887586 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:15.887606 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:15.888065 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:15.889735 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.889740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.889744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.889971 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.890270 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:15.890273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:15.890275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:15.890379 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:15.890860 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:15.890862 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:15.890933 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 16:38:15.891015 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:38:15.891116 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260925 16:38:15.891450 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:15.883627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:52195 (local address 127.0.97.254:44253)
0925 16:38:15.883729 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:15.883733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:15.883746 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:15.883780 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:15.883783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:15.883822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:15.883903 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:15.884676 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.885324 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.886618 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:15.886838 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:15.886915 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:15.888207 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:15.888210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:15.888212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:15.888232 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:15.889627 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.890067 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.890070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.890090 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.890195 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:15.890512 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:15.890516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:15.890517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:15.890765 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:15.890864 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:15.891093 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 16:38:15.891231 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:15.908776 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44253
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:16.022339 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:16.022639 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:16.022719 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:16.026772 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:16.026862 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:16.026937 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:16.027012 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:16.032311 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:16.033495 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:16.034565 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:16.044123 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:16.044129 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:16.044181 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:16.044390 1859 server_base.cc:1034] running on GCE node
I20260925 16:38:16.044793 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:16.045476 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:16.046656 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790354296046631 us; error 30 us; skew 500 ppm
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:16.050124 1859 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:16.050983 1859 webserver.cc:460] Webserver started at http://127.0.97.195:39765/ using document root <none> and password file <none>
I20260925 16:38:16.051573 1859 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:16.051641 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:16.051860 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:16.053882 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "2a4ecb832f774de880e9071e567c5648"
format_stamp: "Formatted at 2026-09-25 16:38:16 on dist-test-slave-kxl8"
server_key: "d2f37c496d048d03f851bfc37e9f8670"
server_key_iv: "ca4803b1d357295a24f7dfeaff4d10ea"
server_key_version: "encryptionkey@0"
I20260925 16:38:16.054414 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "2a4ecb832f774de880e9071e567c5648"
format_stamp: "Formatted at 2026-09-25 16:38:16 on dist-test-slave-kxl8"
server_key: "d2f37c496d048d03f851bfc37e9f8670"
server_key_iv: "ca4803b1d357295a24f7dfeaff4d10ea"
server_key_version: "encryptionkey@0"
I20260925 16:38:16.057931 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:38:16.060240 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:16.061501 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:16.061609 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "2a4ecb832f774de880e9071e567c5648"
format_stamp: "Formatted at 2026-09-25 16:38:16 on dist-test-slave-kxl8"
server_key: "d2f37c496d048d03f851bfc37e9f8670"
server_key_iv: "ca4803b1d357295a24f7dfeaff4d10ea"
server_key_version: "encryptionkey@0"
I20260925 16:38:16.061707 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:16.082309 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:16.085675 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:16.085831 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:16.087162 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:16.087210 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:16.087270 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:16.087303 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:16.097469 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:39411
I20260925 16:38:16.097528 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:39411 every 8 connection(s)
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:16.098424 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:16.101905 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.088061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:41579 (local address 127.0.97.254:44253)
0925 16:38:16.088186 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.088191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.089231 (+ 1040us) server_negotiation.cc:407] Connection header received
0925 16:38:16.090423 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.090430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.090500 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.090604 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.092292 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.093235 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.095118 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.095440 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.095574 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.098430 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:16.098434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:16.098437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:16.098469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:16.100110 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.100566 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.100569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.100570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.100643 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.100945 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.100947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.100948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.101111 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:16.101186 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.101680 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.101788 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:38:16.102631 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.088600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.195:41579)
0925 16:38:16.089083 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.089125 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.090177 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.090798 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.090810 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.091252 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:16.092099 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.092121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.093432 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.093437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.094968 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.094981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.095615 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.095620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.095821 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.096575 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:16.096593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:16.098258 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:16.100224 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.100228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.100239 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.100481 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.100749 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.100751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.100752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.100871 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.101191 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:16.101196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:16.101463 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.101617 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.101865 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260925 16:38:16.103823 1990 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44253
I20260925 16:38:16.103978 1990 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:16.104441 1990 heartbeater.cc:511] Master 127.0.97.254:44253 requested a full tablet report, sending...
I20260925 16:38:16.105105 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 1859
I20260925 16:38:16.105237 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:16.105525 391 external_mini_cluster.cc:1428] Setting key f8d95663472ea729d27b95e954b5ac5a
I20260925 16:38:16.106088 1507 ts_manager.cc:194] Registered new tserver with Master: 2a4ecb832f774de880e9071e567c5648 (127.0.97.195:39411)
I20260925 16:38:16.106829 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:41579
I20260925 16:38:16.108217 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:38:16.113314 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.105986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:43705 (local address 127.0.97.254:44253)
0925 16:38:16.106112 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.106116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.106373 (+ 257us) server_negotiation.cc:407] Connection header received
0925 16:38:16.106485 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.106488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.106531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.106608 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.107020 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.107743 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.108699 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.108921 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.109023 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.110149 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:16.110153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:16.110156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:16.110184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:16.111661 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.112129 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.112133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.112139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.112224 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.112488 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.112493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.112495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.112656 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:16.112754 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.113012 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.113135 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:16.113317 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.105926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.97.195:43705)
0925 16:38:16.106286 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.106302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.106396 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.106639 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.106644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.106810 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:16.106901 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.106905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.107886 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.107889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.108577 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.108585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.109022 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.109026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.109114 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.109604 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:16.109618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:16.109985 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:16.111760 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.111763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.111764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.112018 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.112320 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.112323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.112325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.112406 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.112754 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:16.112757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:16.112851 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.112950 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.113085 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:16.134115 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.126397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.0.97.254:44253)
0925 16:38:16.126517 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.126521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.126533 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:16.126569 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.126571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.126607 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.126719 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.127379 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.127884 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.128715 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.128925 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.129030 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.131243 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:16.131246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:16.131248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:16.131270 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:16.132499 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.132916 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.132925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.132926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.132996 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.133270 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.133304 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.133305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.133468 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:16.133603 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.133842 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.133953 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:16.136813 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:16.139755 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:16.153198 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.147326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42047 (local address 127.0.0.1:48994)
0925 16:38:16.147776 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.147824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.147938 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.148918 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.148924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.148947 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.149078 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.149085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150754 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.150757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.151819 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.151830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.152663 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.152766 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.152801 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.152908 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.153024 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":61,"thread_start_us":80,"threads_started":1}
I20260925 16:38:16.153244 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.148067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39411 (local address 127.0.0.1:33578)
0925 16:38:16.148308 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.148325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.148405 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.149084 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.149087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.149333 (+ 246us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.149409 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.149415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150737 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.150740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.151965 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.151973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.152923 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.153001 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.153020 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.153075 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.153129 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":25,"spinlock_wait_cycles":489856,"thread_start_us":86,"threads_started":1}
I20260925 16:38:16.153393 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.148219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.0.97.195:39411)
0925 16:38:16.148630 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.148640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.148660 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:38:16.148744 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.148749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.148929 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.149158 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.149521 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150638 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152078 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.152817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.153071 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.153123 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.153208 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":100,"threads_started":1}
I20260925 16:38:16.153656 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.147488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.97.193:42047)
0925 16:38:16.148396 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.148411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.148435 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:38:16.148517 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.148522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.148740 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.149245 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.149406 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150591 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.151953 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.152532 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.152809 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.153415 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.153525 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":87,"threads_started":1}
I20260925 16:38:16.153995 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.147546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:37139 (local address 127.0.0.1:57202)
0925 16:38:16.148054 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.148067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.148170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.149084 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.149087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.149116 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.149217 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.149224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150458 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.150464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.151878 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.151889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.153597 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.153600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.153710 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.153735 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.153813 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.153878 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":97,"threads_started":1}
I20260925 16:38:16.154495 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.148364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.97.194:37139)
0925 16:38:16.148675 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.148689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.148707 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:38:16.148766 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.148771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.148928 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.149385 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.149524 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.150352 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152017 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.152819 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.152927 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.153071 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.154255 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.154359 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
I20260925 16:38:16.156217 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 30edddd647734c8fab799067752c5165 (DEFAULT_TABLE table=test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.156405 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 30edddd647734c8fab799067752c5165 (DEFAULT_TABLE table=test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.157224 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edddd647734c8fab799067752c5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.156888 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 30edddd647734c8fab799067752c5165 (DEFAULT_TABLE table=test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.157433 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edddd647734c8fab799067752c5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.157796 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edddd647734c8fab799067752c5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.164429 2013 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Bootstrap starting.
I20260925 16:38:16.165668 2014 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Bootstrap starting.
I20260925 16:38:16.166747 2013 tablet_bootstrap.cc:654] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.167462 2014 tablet_bootstrap.cc:654] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.167841 2013 log.cc:824] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:16.168188 2014 log.cc:824] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:16.168447 2015 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Bootstrap starting.
I20260925 16:38:16.169907 2014 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: No bootstrap required, opened a new log
I20260925 16:38:16.169945 2013 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: No bootstrap required, opened a new log
I20260925 16:38:16.170112 2014 ts_tablet_manager.cc:1397] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:16.170445 2015 tablet_bootstrap.cc:654] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.170274 2013 ts_tablet_manager.cc:1397] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260925 16:38:16.171228 2015 log.cc:824] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:16.173130 2015 tablet_bootstrap.cc:492] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: No bootstrap required, opened a new log
I20260925 16:38:16.173329 2015 ts_tablet_manager.cc:1397] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:38:16.173611 2014 raft_consensus.cc:348] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.173844 2014 raft_consensus.cc:374] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.173893 2014 raft_consensus.cc:729] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4ecb832f774de880e9071e567c5648, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.174312 2014 consensus_queue.cc:260] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.174759 2013 raft_consensus.cc:348] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.175000 2013 raft_consensus.cc:374] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.175058 2013 raft_consensus.cc:729] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfee1a21264e460793fa2f43e98bddde, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.175180 2014 ts_tablet_manager.cc:1428] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:38:16.175457 2013 consensus_queue.cc:260] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.175526 1990 heartbeater.cc:503] Master 127.0.97.254:44253 was elected leader, sending a full tablet report...
I20260925 16:38:16.176020 1710 heartbeater.cc:503] Master 127.0.97.254:44253 was elected leader, sending a full tablet report...
I20260925 16:38:16.176352 2013 ts_tablet_manager.cc:1428] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 16:38:16.177026 2015 raft_consensus.cc:348] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.177253 2015 raft_consensus.cc:374] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.177335 2015 raft_consensus.cc:729] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb2d43d006d4e22b57d5ab8667793bd, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.177675 2015 consensus_queue.cc:260] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.178201 1850 heartbeater.cc:503] Master 127.0.97.254:44253 was elected leader, sending a full tablet report...
I20260925 16:38:16.178488 2015 ts_tablet_manager.cc:1428] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 16:38:16.349449 1991 tablet.cc:2369] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:38:16.377667 1851 tablet.cc:2369] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:38:16.398042 1711 tablet.cc:2369] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:16.461603 2021 raft_consensus.cc:482] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:16.461867 2021 raft_consensus.cc:504] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.463115 2021 leader_election.cc:288] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 2a4ecb832f774de880e9071e567c5648 (127.0.97.195:39411)
I20260925 16:38:16.467263 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.463434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42047 (local address 127.0.97.194:54259)
0925 16:38:16.463668 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.463685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.463847 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.464185 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.464189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.464208 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.464331 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.464336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.465138 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.465143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.466135 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.466144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.466806 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.466809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.466910 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.466935 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.467020 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.467098 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260925 16:38:16.467399 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.463525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:54259 (local address 127.0.97.193:42047)
0925 16:38:16.463667 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.463671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.463775 (+ 104us) server_negotiation.cc:407] Connection header received
0925 16:38:16.463978 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.463982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.464037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.464158 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.464439 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.465031 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.466278 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.466676 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.466805 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.466860 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.467176 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.467272 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:16.467519 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.463864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39411 (local address 127.0.97.194:40665)
0925 16:38:16.464184 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.464198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.464315 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.464578 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.464581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.464596 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.464720 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.464725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.465556 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.465562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.466421 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.466430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.467043 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.467046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.467147 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.467168 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.467309 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.467381 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":81,"threads_started":1}
I20260925 16:38:16.467509 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.464031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:40665 (local address 127.0.97.195:39411)
0925 16:38:16.464188 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.464195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.464255 (+ 60us) server_negotiation.cc:407] Connection header received
0925 16:38:16.464426 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.464429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.464478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.464557 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.464897 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.465451 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.466535 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.466925 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.467049 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.467095 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.467328 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.467394 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:38:16.468057 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edddd647734c8fab799067752c5165" candidate_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4ecb832f774de880e9071e567c5648" is_pre_election: true
I20260925 16:38:16.468042 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edddd647734c8fab799067752c5165" candidate_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde" is_pre_election: true
I20260925 16:38:16.468387 1664 raft_consensus.cc:2454] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb2d43d006d4e22b57d5ab8667793bd in term 0.
I20260925 16:38:16.468387 1944 raft_consensus.cc:2454] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb2d43d006d4e22b57d5ab8667793bd in term 0.
I20260925 16:38:16.468945 1736 leader_election.cc:302] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2d43d006d4e22b57d5ab8667793bd, bfee1a21264e460793fa2f43e98bddde; no voters:
I20260925 16:38:16.469202 2021 raft_consensus.cc:2781] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:16.469321 2021 raft_consensus.cc:482] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:16.469376 2021 raft_consensus.cc:3037] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.470422 2021 raft_consensus.cc:504] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.470773 2021 leader_election.cc:288] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [CANDIDATE]: Term 1 election: Requested vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 2a4ecb832f774de880e9071e567c5648 (127.0.97.195:39411)
I20260925 16:38:16.471141 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edddd647734c8fab799067752c5165" candidate_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde"
I20260925 16:38:16.471220 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edddd647734c8fab799067752c5165" candidate_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4ecb832f774de880e9071e567c5648"
I20260925 16:38:16.471293 1664 raft_consensus.cc:3037] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.471359 1944 raft_consensus.cc:3037] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.472388 1664 raft_consensus.cc:2454] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2d43d006d4e22b57d5ab8667793bd in term 1.
I20260925 16:38:16.472741 1736 leader_election.cc:302] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2d43d006d4e22b57d5ab8667793bd, bfee1a21264e460793fa2f43e98bddde; no voters:
I20260925 16:38:16.472818 1944 raft_consensus.cc:2454] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2d43d006d4e22b57d5ab8667793bd in term 1.
I20260925 16:38:16.472956 2021 raft_consensus.cc:2781] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:16.473166 2021 raft_consensus.cc:686] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 LEADER]: Becoming Leader. State: Replica: 7fb2d43d006d4e22b57d5ab8667793bd, State: Running, Role: LEADER
I20260925 16:38:16.473547 2021 consensus_queue.cc:237] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.476802 1505 catalog_manager.cc:5495] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194). New cstate: current_term: 1 leader_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:16.527878 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.524422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.0.97.194:37139)
0925 16:38:16.524542 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.524545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.524560 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:16.524816 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.524819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.524875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.524961 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:16.525174 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.525779 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.526609 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.526814 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.526881 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.527148 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.527536 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.527619 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.527718 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.527767 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:38:16.533829 1944 raft_consensus.cc:1261] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Refusing update from remote peer 7fb2d43d006d4e22b57d5ab8667793bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.533829 1664 raft_consensus.cc:1261] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Refusing update from remote peer 7fb2d43d006d4e22b57d5ab8667793bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.534477 2021 consensus_queue.cc:1035] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.534734 2024 consensus_queue.cc:1035] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.541522 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333291198594985984 to 7333291198358220800. Current Snapshot: MvccSnapshot[applied={T|T < 7333291198594985984}]
I20260925 16:38:16.551910 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:16.552080 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:16.554167 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Sending DeleteTablet for 3 replicas of tablet 30edddd647734c8fab799067752c5165
I20260925 16:38:16.554884 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 30edddd647734c8fab799067752c5165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:57202
I20260925 16:38:16.554944 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 30edddd647734c8fab799067752c5165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:48994
I20260925 16:38:16.555045 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 30edddd647734c8fab799067752c5165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:33578
I20260925 16:38:16.555519 2038 tablet_replica.cc:330] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: stopping tablet replica
I20260925 16:38:16.555610 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:16.555814 2038 raft_consensus.cc:2229] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 16:38:16.556003 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:16.556092 2038 raft_consensus.cc:2258] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.556442 2037 tablet_replica.cc:330] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: stopping tablet replica
I20260925 16:38:16.556704 2037 raft_consensus.cc:2229] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:16.556949 2037 raft_consensus.cc:2258] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.557374 2036 tablet_replica.cc:330] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: stopping tablet replica
I20260925 16:38:16.557615 2036 raft_consensus.cc:2229] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:16.557924 2036 raft_consensus.cc:2258] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.558835 2038 ts_tablet_manager.cc:1905] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.559403 2036 ts_tablet_manager.cc:1905] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.560570 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd (DEFAULT_TABLE table=test-table [id=3ad1aef5aad049b8a19761ec93d4079f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.560756 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd (DEFAULT_TABLE table=test-table [id=3ad1aef5aad049b8a19761ec93d4079f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.560839 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45d47ae63a4a9ca88d5f44124890fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.560815 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd (DEFAULT_TABLE table=test-table [id=3ad1aef5aad049b8a19761ec93d4079f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:16.560963 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45d47ae63a4a9ca88d5f44124890fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.561050 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45d47ae63a4a9ca88d5f44124890fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.563244 2013 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Bootstrap starting.
I20260925 16:38:16.563624 2015 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Bootstrap starting.
I20260925 16:38:16.563696 2014 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Bootstrap starting.
I20260925 16:38:16.563946 2036 ts_tablet_manager.cc:1918] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:16.564038 2036 log.cc:1196] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/30edddd647734c8fab799067752c5165
I20260925 16:38:16.564204 2013 tablet_bootstrap.cc:654] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.564366 2036 ts_tablet_manager.cc:1939] T 30edddd647734c8fab799067752c5165 P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting consensus metadata
I20260925 16:38:16.564559 2015 tablet_bootstrap.cc:654] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.564733 2014 tablet_bootstrap.cc:654] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.565567 1494 catalog_manager.cc:4841] TS 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139): tablet 30edddd647734c8fab799067752c5165 (table test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]) successfully deleted
I20260925 16:38:16.566227 2014 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: No bootstrap required, opened a new log
I20260925 16:38:16.566318 2014 ts_tablet_manager.cc:1397] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:16.566498 2015 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: No bootstrap required, opened a new log
I20260925 16:38:16.566599 2015 ts_tablet_manager.cc:1397] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:16.566797 2014 raft_consensus.cc:348] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.566917 2014 raft_consensus.cc:374] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.566951 2014 raft_consensus.cc:729] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4ecb832f774de880e9071e567c5648, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.567015 2015 raft_consensus.cc:348] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.567075 2014 consensus_queue.cc:260] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.567129 2015 raft_consensus.cc:374] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.567165 2015 raft_consensus.cc:729] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb2d43d006d4e22b57d5ab8667793bd, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.567291 2015 consensus_queue.cc:260] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.567497 2014 ts_tablet_manager.cc:1428] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:16.567528 2015 ts_tablet_manager.cc:1428] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:16.566056 2037 ts_tablet_manager.cc:1905] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.566737 2013 tablet_bootstrap.cc:492] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: No bootstrap required, opened a new log
I20260925 16:38:16.568125 2013 ts_tablet_manager.cc:1397] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260925 16:38:16.568585 2013 raft_consensus.cc:348] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.568691 2013 raft_consensus.cc:374] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.568725 2013 raft_consensus.cc:729] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfee1a21264e460793fa2f43e98bddde, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.568840 2013 consensus_queue.cc:260] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.569114 2013 ts_tablet_manager.cc:1428] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:16.570181 2038 ts_tablet_manager.cc:1918] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:16.570261 2038 log.cc:1196] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/30edddd647734c8fab799067752c5165
I20260925 16:38:16.570547 2038 ts_tablet_manager.cc:1939] T 30edddd647734c8fab799067752c5165 P 2a4ecb832f774de880e9071e567c5648: Deleting consensus metadata
I20260925 16:38:16.571832 1492 catalog_manager.cc:4841] TS 2a4ecb832f774de880e9071e567c5648 (127.0.97.195:39411): tablet 30edddd647734c8fab799067752c5165 (table test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]) successfully deleted
I20260925 16:38:16.574211 2037 ts_tablet_manager.cc:1918] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:16.574317 2037 log.cc:1196] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/30edddd647734c8fab799067752c5165
I20260925 16:38:16.574671 2037 ts_tablet_manager.cc:1939] T 30edddd647734c8fab799067752c5165 P bfee1a21264e460793fa2f43e98bddde: Deleting consensus metadata
I20260925 16:38:16.575588 1492 catalog_manager.cc:4841] TS bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047): tablet 30edddd647734c8fab799067752c5165 (table test-table [id=0c7ad0799a8c41d2a218628c0df56e9f]) successfully deleted
I20260925 16:38:16.716672 2019 raft_consensus.cc:482] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:16.716825 2019 raft_consensus.cc:504] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.717856 2019 leader_election.cc:288] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139)
I20260925 16:38:16.723125 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.718233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:45801 (local address 127.0.97.194:37139)
0925 16:38:16.718375 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.718380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.718481 (+ 101us) server_negotiation.cc:407] Connection header received
0925 16:38:16.718641 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.718644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.718693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.718783 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.719234 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.720006 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.721891 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.722490 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.722614 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.722666 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.722923 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.722994 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:38:16.723125 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.718201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:44327 (local address 127.0.97.193:42047)
0925 16:38:16.718419 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.718423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.718513 (+ 90us) server_negotiation.cc:407] Connection header received
0925 16:38:16.718641 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.718644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.718722 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.718800 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:16.719233 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.720023 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.721890 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.722490 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.722614 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.722666 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.722923 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.722994 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 16:38:16.723217 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.718097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:37139 (local address 127.0.97.195:45801)
0925 16:38:16.718375 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.718391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.718516 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.718829 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.718832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.718876 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.719120 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.719127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.720142 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.720146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.721761 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.721773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.722616 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.722620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.722746 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.722815 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.722897 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.723032 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":307,"spinlock_wait_cycles":38144,"thread_start_us":86,"threads_started":1}
I20260925 16:38:16.723585 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.718099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42047 (local address 127.0.97.195:44327)
0925 16:38:16.718430 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.718446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.718539 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.718829 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.718832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.718877 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:16.719121 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.719127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.720142 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.720146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.721761 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.721773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.722616 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.722620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.722759 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.722815 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.722897 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.723032 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":245,"spinlock_wait_cycles":9088,"thread_start_us":75,"threads_started":1}
I20260925 16:38:16.723788 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45d47ae63a4a9ca88d5f44124890fd" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" is_pre_election: true
I20260925 16:38:16.724009 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45d47ae63a4a9ca88d5f44124890fd" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde" is_pre_election: true
I20260925 16:38:16.724092 1804 raft_consensus.cc:2454] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 0.
I20260925 16:38:16.724165 1664 raft_consensus.cc:2454] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 0.
I20260925 16:38:16.724532 1876 leader_election.cc:302] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4ecb832f774de880e9071e567c5648, bfee1a21264e460793fa2f43e98bddde; no voters:
I20260925 16:38:16.724845 2019 raft_consensus.cc:2781] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:16.724916 2019 raft_consensus.cc:482] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:16.724992 2019 raft_consensus.cc:3037] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.726107 2019 raft_consensus.cc:504] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.726447 2019 leader_election.cc:288] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 election: Requested vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139)
I20260925 16:38:16.726759 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45d47ae63a4a9ca88d5f44124890fd" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde"
I20260925 16:38:16.726898 1664 raft_consensus.cc:3037] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.726917 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45d47ae63a4a9ca88d5f44124890fd" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2d43d006d4e22b57d5ab8667793bd"
I20260925 16:38:16.727029 1804 raft_consensus.cc:3037] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.728036 1664 raft_consensus.cc:2454] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 1.
I20260925 16:38:16.728165 1804 raft_consensus.cc:2454] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 1.
I20260925 16:38:16.728332 1876 leader_election.cc:302] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4ecb832f774de880e9071e567c5648, bfee1a21264e460793fa2f43e98bddde; no voters:
I20260925 16:38:16.728538 2019 raft_consensus.cc:2781] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:16.728725 2019 raft_consensus.cc:686] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 1 LEADER]: Becoming Leader. State: Replica: 2a4ecb832f774de880e9071e567c5648, State: Running, Role: LEADER
I20260925 16:38:16.728969 2019 consensus_queue.cc:237] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.731185 1505 catalog_manager.cc:5495] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a4ecb832f774de880e9071e567c5648 (127.0.97.195). New cstate: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } health_report { overall_health: HEALTHY } } }
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:16.782925 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.774970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44253 (local address 127.0.0.1:40088)
0925 16:38:16.775329 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.775340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.775443 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.775671 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.775689 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.776109 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:16.776211 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.776217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.777012 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.777015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.777577 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.777584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.778007 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.778009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.778065 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.778631 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:16.778643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:16.780015 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:16.781613 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.781620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.781624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.781843 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.782072 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:16.782074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:16.782075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:16.782146 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:16.782451 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:16.782455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:16.782548 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.782633 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.782759 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260925 16:38:16.783083 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.775004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.97.254:44253)
0925 16:38:16.775236 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.775244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.775414 (+ 170us) server_negotiation.cc:407] Connection header received
0925 16:38:16.775520 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.775522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.775564 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.775642 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.776361 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.776919 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.777690 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.777915 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.777992 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.780133 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:16.780136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:16.780138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:16.780178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:16.781513 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.781921 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.781924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.781925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.781998 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.782216 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.782218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.782219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.782371 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:16.782444 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.782690 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.782852 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260925 16:38:16.786748 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:40088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:38:16.787420 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:16.791816 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaddccf94841abba1d0d753a60246c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6de415cb9af047a0901de457f009f5fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:16.791889 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaddccf94841abba1d0d753a60246c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6de415cb9af047a0901de457f009f5fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:16.792210 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16aaddccf94841abba1d0d753a60246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.792277 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16aaddccf94841abba1d0d753a60246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.792171 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaddccf94841abba1d0d753a60246c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6de415cb9af047a0901de457f009f5fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:16.792481 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16aaddccf94841abba1d0d753a60246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:16.794652 2015 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: Bootstrap starting.
I20260925 16:38:16.795301 2014 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: Bootstrap starting.
I20260925 16:38:16.795305 2013 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: Bootstrap starting.
I20260925 16:38:16.795598 2015 tablet_bootstrap.cc:654] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.796366 2014 tablet_bootstrap.cc:654] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.796743 2013 tablet_bootstrap.cc:654] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:16.797089 2015 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: No bootstrap required, opened a new log
I20260925 16:38:16.797158 2015 ts_tablet_manager.cc:1397] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:16.797657 2014 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: No bootstrap required, opened a new log
I20260925 16:38:16.797731 2014 ts_tablet_manager.cc:1397] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:16.797658 2015 raft_consensus.cc:348] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.797770 2015 raft_consensus.cc:374] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.797806 2015 raft_consensus.cc:729] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb2d43d006d4e22b57d5ab8667793bd, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.797940 2015 consensus_queue.cc:260] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.797998 2013 tablet_bootstrap.cc:492] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: No bootstrap required, opened a new log
I20260925 16:38:16.798091 2013 ts_tablet_manager.cc:1397] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:38:16.798153 2014 raft_consensus.cc:348] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.798257 2014 raft_consensus.cc:374] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.798208 2024 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.798305 2014 raft_consensus.cc:729] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4ecb832f774de880e9071e567c5648, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.798349 2024 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.798420 2014 consensus_queue.cc:260] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.798221 2015 ts_tablet_manager.cc:1428] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:16.798667 2013 raft_consensus.cc:348] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.798823 2013 raft_consensus.cc:374] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:16.798864 2013 raft_consensus.cc:729] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfee1a21264e460793fa2f43e98bddde, State: Initialized, Role: FOLLOWER
I20260925 16:38:16.798919 2019 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.799046 2019 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.799005 2013 consensus_queue.cc:260] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.799412 2014 ts_tablet_manager.cc:1428] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:16.799702 2013 ts_tablet_manager.cc:1428] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 16:38:16.799656 2020 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.799779 2020 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.870095 2019 raft_consensus.cc:482] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:16.870226 2019 raft_consensus.cc:504] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.870620 2019 leader_election.cc:288] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139)
I20260925 16:38:16.871102 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16aaddccf94841abba1d0d753a60246c" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde" is_pre_election: true
I20260925 16:38:16.871102 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16aaddccf94841abba1d0d753a60246c" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" is_pre_election: true
I20260925 16:38:16.871263 1804 raft_consensus.cc:2454] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 0.
I20260925 16:38:16.871263 1664 raft_consensus.cc:2454] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 0.
I20260925 16:38:16.871562 1876 leader_election.cc:302] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4ecb832f774de880e9071e567c5648, bfee1a21264e460793fa2f43e98bddde; no voters:
I20260925 16:38:16.871769 2019 raft_consensus.cc:2781] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:16.871837 2019 raft_consensus.cc:482] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:16.871877 2019 raft_consensus.cc:3037] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.872859 2019 raft_consensus.cc:504] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.873122 2019 leader_election.cc:288] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 election: Requested vote from peers bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047), 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139)
I20260925 16:38:16.873476 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16aaddccf94841abba1d0d753a60246c" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2d43d006d4e22b57d5ab8667793bd"
I20260925 16:38:16.873476 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16aaddccf94841abba1d0d753a60246c" candidate_uuid: "2a4ecb832f774de880e9071e567c5648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfee1a21264e460793fa2f43e98bddde"
I20260925 16:38:16.873608 1664 raft_consensus.cc:3037] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.873602 1804 raft_consensus.cc:3037] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:16.874660 1664 raft_consensus.cc:2454] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 1.
I20260925 16:38:16.874660 1804 raft_consensus.cc:2454] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4ecb832f774de880e9071e567c5648 in term 1.
I20260925 16:38:16.874948 1878 leader_election.cc:302] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4ecb832f774de880e9071e567c5648, 7fb2d43d006d4e22b57d5ab8667793bd; no voters:
I20260925 16:38:16.875177 2019 raft_consensus.cc:2781] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:16.875274 2019 raft_consensus.cc:686] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [term 1 LEADER]: Becoming Leader. State: Replica: 2a4ecb832f774de880e9071e567c5648, State: Running, Role: LEADER
I20260925 16:38:16.875398 2019 consensus_queue.cc:237] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } }
I20260925 16:38:16.876120 2044 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: TxnStatusTablet state changed. Reason: New leader 2a4ecb832f774de880e9071e567c5648. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.876227 2044 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:16.876487 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:38:16.876623 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:38:16.877079 1506 catalog_manager.cc:5495] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a4ecb832f774de880e9071e567c5648 (127.0.97.195). New cstate: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } health_report { overall_health: HEALTHY } } }
I20260925 16:38:16.899920 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.896263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39411 (local address 127.0.0.1:33592)
0925 16:38:16.896489 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.896506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.896616 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.896936 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.896939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.896948 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:16.897050 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.897056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.897823 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.897827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.898430 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.898438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.898875 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.898878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.898977 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.899022 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.899637 (+ 615us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.899652 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.899714 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.899794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 16:38:16.899920 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.896360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.0.97.195:39411)
0925 16:38:16.896488 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.896492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.896554 (+ 62us) server_negotiation.cc:407] Connection header received
0925 16:38:16.896761 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.896764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.896820 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.896912 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:16.897157 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.897717 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.898545 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.898773 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.898875 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.899157 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.899543 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.899661 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.899735 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.899786 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:38:16.902737 1664 raft_consensus.cc:1261] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Refusing update from remote peer 2a4ecb832f774de880e9071e567c5648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.902936 1804 raft_consensus.cc:1261] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Refusing update from remote peer 2a4ecb832f774de880e9071e567c5648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.903277 2044 consensus_queue.cc:1035] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.903450 2019 consensus_queue.cc:1035] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.904732 2020 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: TxnStatusTablet state changed. Reason: New leader 2a4ecb832f774de880e9071e567c5648. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.904850 2020 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.904878 2024 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: TxnStatusTablet state changed. Reason: New leader 2a4ecb832f774de880e9071e567c5648. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.904996 2024 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.908384 2024 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.908485 2024 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 7fb2d43d006d4e22b57d5ab8667793bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.908926 2019 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.909029 2019 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:16.909536 2020 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.909633 2020 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P bfee1a21264e460793fa2f43e98bddde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:16.913141 2062 tablet_replica.cc:439] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a4ecb832f774de880e9071e567c5648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 } } peers { permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 } } peers { permanent_uuid: "2a4ecb832f774de880e9071e567c5648" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39411 } } }
I20260925 16:38:16.913262 2062 tablet_replica.cc:442] T 16aaddccf94841abba1d0d753a60246c P 2a4ecb832f774de880e9071e567c5648: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:16.916481 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.913188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.97.195:39411)
0925 16:38:16.913311 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.913317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.913335 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:38:16.913492 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.913495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.913536 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.913608 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:16.913869 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.914623 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.915428 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.915803 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.915905 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.916090 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.916166 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.916230 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.916310 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.916358 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:38:16.926851 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.923023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:46557 (local address 127.0.97.195:39411)
0925 16:38:16.923124 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.923133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.923771 (+ 638us) server_negotiation.cc:407] Connection header received
0925 16:38:16.923908 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.923912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.923953 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.924024 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:16.924250 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.924796 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.925741 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.926095 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.926202 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.926418 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.926490 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.926551 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.926648 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.926699 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 16:38:16.926860 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.922967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39411 (local address 127.0.97.195:46557)
0925 16:38:16.923694 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:16.923707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:16.923822 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:16.924028 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:16.924031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:16.924037 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:16.924157 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.924161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.924895 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.924898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.925603 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:16.925611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.926196 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.926198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:16.926265 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.926305 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:16.926568 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:16.926572 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:38:16.926616 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 16:38:16.926669 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":576,"threads_started":1}
I20260925 16:38:16.932880 1804 raft_consensus.cc:1261] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Refusing update from remote peer 2a4ecb832f774de880e9071e567c5648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.932973 1664 raft_consensus.cc:1261] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Refusing update from remote peer 2a4ecb832f774de880e9071e567c5648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:16.933373 2019 consensus_queue.cc:1035] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfee1a21264e460793fa2f43e98bddde" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.933512 2062 consensus_queue.cc:1035] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2d43d006d4e22b57d5ab8667793bd" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:16.938827 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333291200233533440 to 7333291199403704320. Current Snapshot: MvccSnapshot[applied={T|T < 7333291200233533440}]
I20260925 16:38:16.976837 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:16.977012 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:16.979007 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cc817064bf34ffab96d07dd92ad0294: Sending DeleteTablet for 3 replicas of tablet 8a45d47ae63a4a9ca88d5f44124890fd
I20260925 16:38:16.979568 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:57202
I20260925 16:38:16.979576 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:48994
I20260925 16:38:16.979773 2036 tablet_replica.cc:330] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: stopping tablet replica
I20260925 16:38:16.979802 2037 tablet_replica.cc:330] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: stopping tablet replica
I20260925 16:38:16.979897 2037 raft_consensus.cc:2229] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:16.979897 2036 raft_consensus.cc:2229] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:16.980014 2037 raft_consensus.cc:2258] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.980014 2036 raft_consensus.cc:2258] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.980388 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45d47ae63a4a9ca88d5f44124890fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:33578
I20260925 16:38:16.980554 2038 tablet_replica.cc:330] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: stopping tablet replica
I20260925 16:38:16.980647 2038 raft_consensus.cc:2229] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:16.980741 2037 ts_tablet_manager.cc:1905] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.980939 2038 raft_consensus.cc:2258] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:16.981698 2038 ts_tablet_manager.cc:1905] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.983983 2038 ts_tablet_manager.cc:1918] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:16.984071 2038 log.cc:1196] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/8a45d47ae63a4a9ca88d5f44124890fd
I20260925 16:38:16.984180 2036 ts_tablet_manager.cc:1905] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:16.984365 2038 ts_tablet_manager.cc:1939] T 8a45d47ae63a4a9ca88d5f44124890fd P 2a4ecb832f774de880e9071e567c5648: Deleting consensus metadata
I20260925 16:38:16.984988 1492 catalog_manager.cc:4841] TS 2a4ecb832f774de880e9071e567c5648 (127.0.97.195:39411): tablet 8a45d47ae63a4a9ca88d5f44124890fd (table test-table [id=3ad1aef5aad049b8a19761ec93d4079f]) successfully deleted
I20260925 16:38:16.985613 2037 ts_tablet_manager.cc:1918] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:16.985713 2037 log.cc:1196] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/8a45d47ae63a4a9ca88d5f44124890fd
I20260925 16:38:16.985940 2037 ts_tablet_manager.cc:1939] T 8a45d47ae63a4a9ca88d5f44124890fd P bfee1a21264e460793fa2f43e98bddde: Deleting consensus metadata
I20260925 16:38:16.986572 1492 catalog_manager.cc:4841] TS bfee1a21264e460793fa2f43e98bddde (127.0.97.193:42047): tablet 8a45d47ae63a4a9ca88d5f44124890fd (table test-table [id=3ad1aef5aad049b8a19761ec93d4079f]) successfully deleted
I20260925 16:38:16.986873 2036 ts_tablet_manager.cc:1918] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:16.986951 2036 log.cc:1196] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/8a45d47ae63a4a9ca88d5f44124890fd
I20260925 16:38:16.987200 2036 ts_tablet_manager.cc:1939] T 8a45d47ae63a4a9ca88d5f44124890fd P 7fb2d43d006d4e22b57d5ab8667793bd: Deleting consensus metadata
I20260925 16:38:16.987850 1494 catalog_manager.cc:4841] TS 7fb2d43d006d4e22b57d5ab8667793bd (127.0.97.194:37139): tablet 8a45d47ae63a4a9ca88d5f44124890fd (table test-table [id=3ad1aef5aad049b8a19761ec93d4079f]) successfully deleted
Sep 25 16:38:16 dist-test-slave-kxl8 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:16.996580 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.983043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.97.193:42047)
0925 16:38:16.983204 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.983209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.986718 (+ 3509us) server_negotiation.cc:407] Connection header received
0925 16:38:16.986896 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.986899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.986947 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.987020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.987861 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.988668 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.989675 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:16.989896 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:16.990001 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:16.992975 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:16.992986 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:16.993001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:16.993028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:16.994930 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.995340 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.995347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.995349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.995443 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:16.995691 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:16.995694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:16.995696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:16.995977 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:16.996174 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:16.996306 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 16:38:16.996427 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 16:38:16.997114 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49006
I20260925 16:38:17.006930 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:16.998849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.97.254:44253)
0925 16:38:16.998978 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:16.998983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:16.999019 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:38:16.999126 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:16.999129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:16.999174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:16.999253 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:16.999796 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:17.001048 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:17.001867 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:17.002229 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:17.002339 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:17.003623 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:17.003627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:17.003630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:17.003692 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:17.005245 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:17.005727 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:17.005730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:17.005732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:17.005821 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:17.006061 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:17.006064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:17.006066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:17.006293 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:17.006434 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:17.006637 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:38:17.006774 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 16:38:17.007359 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40092
I20260925 16:38:17.008569 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1579
I20260925 16:38:17.017153 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1719
I20260925 16:38:17.026062 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1859
I20260925 16:38:17.033097 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 1475
2026-09-25T16:38:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3036 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:17 dist-test-slave-kxl8 krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:17 dist-test-slave-kxl8 krb5kdc[2076](info): set up 2 sockets
Sep 25 16:38:17 dist-test-slave-kxl8 krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260925 16:38:18.085362 391 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 16:38:18.194801 391 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.109s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 16:38:18.355451 391 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.161s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 16:38:18.475251 391 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.120s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:18 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:18.538223 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45867
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:18.646929 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:18.647184 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:18.647222 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:18.651202 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:18.651258 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:38:18.651275 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:18.651292 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:18.651341 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:18.651371 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:18.656591 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:45867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:18.657667 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:18.658675 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:18.667142 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:18.667166 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:18.667179 2089 server_base.cc:1034] running on GCE node
W20260925 16:38:18.667266 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:18.667927 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:18.669008 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:18.670164 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790354298670140 us; error 38 us; skew 500 ppm
Sep 25 16:38:18 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:18.672904 2089 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:18.673987 2089 webserver.cc:460] Webserver started at http://127.0.97.254:33835/ using document root <none> and password file <none>
I20260925 16:38:18.674626 2089 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:18.674693 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:18.674921 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:18.676929 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "94c223d1cf1f45e1ba928058fa480597"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "e0255a0fc39e87c9dacd67d648f25ab4"
server_key_iv: "31409e8b579f0726618a725664b28999"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.677513 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "94c223d1cf1f45e1ba928058fa480597"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "e0255a0fc39e87c9dacd67d648f25ab4"
server_key_iv: "31409e8b579f0726618a725664b28999"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.681246 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:38:18.683650 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:18.684891 2089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:18.685030 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "94c223d1cf1f45e1ba928058fa480597"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "e0255a0fc39e87c9dacd67d648f25ab4"
server_key_iv: "31409e8b579f0726618a725664b28999"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.685168 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:18.696530 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:18.699292 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:18.699498 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:18.707535 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:45867
I20260925 16:38:18.707547 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:45867 every 8 connection(s)
I20260925 16:38:18.708482 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:18.711336 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:18.714277 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2089
I20260925 16:38:18.714406 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:18.714660 391 external_mini_cluster.cc:1428] Setting key ca0f7025e9b4ade3f0e74dfc62d8709e
I20260925 16:38:18.717460 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Bootstrap starting.
I20260925 16:38:18.720003 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:18.720872 2167 log.cc:824] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:18.722756 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: No bootstrap required, opened a new log
Sep 25 16:38:18 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:18.726558 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } }
I20260925 16:38:18.726861 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:18.726980 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c223d1cf1f45e1ba928058fa480597, State: Initialized, Role: FOLLOWER
I20260925 16:38:18.727469 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } }
I20260925 16:38:18.727595 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:18.727684 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:18.727818 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:18.728878 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } }
I20260925 16:38:18.729223 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c223d1cf1f45e1ba928058fa480597; no voters:
I20260925 16:38:18.729461 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:18.716055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.0.97.254:45867)
0925 16:38:18.716470 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:18.716485 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:18.716520 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:38:18.717176 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:18.717194 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:18.717642 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:18.717918 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:18.718781 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:18.719786 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:18.720984 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:18.721556 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:18.721724 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:18.724485 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:18.724490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:18.724503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:18.724528 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:18.726313 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:18.726811 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:18.726815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:18.726819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:18.726996 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:18.727313 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:18.727316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:18.727318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:18.727810 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:18.728024 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:18.728273 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 16:38:18.728578 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260925 16:38:18.729592 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:18.729697 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:18.729867 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [term 1 LEADER]: Becoming Leader. State: Replica: 94c223d1cf1f45e1ba928058fa480597, State: Running, Role: LEADER
I20260925 16:38:18.730140 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } }
I20260925 16:38:18.730846 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:18.732689 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c223d1cf1f45e1ba928058fa480597. Latest consensus state: current_term: 1 leader_uuid: "94c223d1cf1f45e1ba928058fa480597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } } }
I20260925 16:38:18.732805 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:18.733119 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c223d1cf1f45e1ba928058fa480597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c223d1cf1f45e1ba928058fa480597" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45867 } } }
I20260925 16:38:18.733445 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597 [sys.catalog]: This master's current role is: LEADER
W20260925 16:38:18.735400 2187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:38:18.735491 2187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:38:18.735564 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:18.736511 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:18.742516 2188 catalog_manager.cc:1347] Generated new cluster ID: e66a764564964351b08229ce7ad853ef
I20260925 16:38:18.742589 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:18.758706 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:18.759719 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:18.766690 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Generated new TSK 0
I20260925 16:38:18.767438 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:18.803944 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45867
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:18.942039 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:18.942296 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:18.942342 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:18.946317 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:18.946375 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:18.946416 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:38:18.946430 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:18.946462 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:18.951582 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:18.952528 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:18.953493 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:18.962910 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:18.962899 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:18.962904 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:18.963267 2193 server_base.cc:1034] running on GCE node
I20260925 16:38:18.963672 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:18.964350 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:18.965530 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790354298965508 us; error 44 us; skew 500 ppm
Sep 25 16:38:18 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:18.968231 2193 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:18.969054 2193 webserver.cc:460] Webserver started at http://127.0.97.193:43177/ using document root <none> and password file <none>
I20260925 16:38:18.969651 2193 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:18.969719 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:18.969887 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:18.971944 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "328f1a52705e4f5181f59ef3115e5739"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "329ec06e4d78235df8334d0d3aafe6d0"
server_key_iv: "e8c770dda20207de6f9fbd6c8e1eabf5"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.972468 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "328f1a52705e4f5181f59ef3115e5739"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "329ec06e4d78235df8334d0d3aafe6d0"
server_key_iv: "e8c770dda20207de6f9fbd6c8e1eabf5"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.976589 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:38:18.979090 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:18.980436 2193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:18.980551 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "328f1a52705e4f5181f59ef3115e5739"
format_stamp: "Formatted at 2026-09-25 16:38:18 on dist-test-slave-kxl8"
server_key: "329ec06e4d78235df8334d0d3aafe6d0"
server_key_iv: "e8c770dda20207de6f9fbd6c8e1eabf5"
server_key_version: "encryptionkey@0"
I20260925 16:38:18.980665 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:18.995086 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:18.998014 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:18.998219 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:18.999593 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:18.999637 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:18.999706 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:18.999744 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:19.010711 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:43105
I20260925 16:38:19.010833 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:43105 every 8 connection(s)
I20260925 16:38:19.011904 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:19.014474 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.000791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:59735 (local address 127.0.97.254:45867)
0925 16:38:19.000922 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.000926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.001946 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 16:38:19.003221 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.003226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.003295 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.003405 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.004976 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.005930 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.007321 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.007581 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.007697 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.010297 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.010301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.010304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.010336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.012080 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.012711 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.012715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.012717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.012829 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.013237 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.013240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.013241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.013545 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.013647 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.014191 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.014319 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:19.015220 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.001261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.193:59735)
0925 16:38:19.001792 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.001834 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.002923 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.003612 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.003623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.004105 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.004814 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.004832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.006068 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.006073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.007193 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.007202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.007732 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.007736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.007916 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.008538 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.008572 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.010111 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.012221 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.012226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.012237 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.012580 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.012942 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.012946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.012948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.013105 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.013701 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.013706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.013921 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.014095 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.014343 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260925 16:38:19.016716 2324 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45867
I20260925 16:38:19.016883 2324 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:19.017490 2324 heartbeater.cc:511] Master 127.0.97.254:45867 requested a full tablet report, sending...
I20260925 16:38:19.019143 2120 ts_manager.cc:194] Registered new tserver with Master: 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105)
I20260925 16:38:19.020689 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:59735
I20260925 16:38:19.021090 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2193
I20260925 16:38:19.021216 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:19.021584 391 external_mini_cluster.cc:1428] Setting key 18b4ea4467520977d21967271085ccfa
I20260925 16:38:19.028458 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.019038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:49441 (local address 127.0.97.254:45867)
0925 16:38:19.019163 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.019169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.019313 (+ 144us) server_negotiation.cc:407] Connection header received
0925 16:38:19.019445 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.019447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.019489 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.019570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.019977 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.020771 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.021909 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.022214 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.022316 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.024878 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.024882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.024885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.024909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.026807 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.027233 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.027238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.027240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.027325 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.027605 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.027610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.027612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.027796 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.027899 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.028177 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.028298 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":49}
I20260925 16:38:19.028463 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.018888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.193:49441)
0925 16:38:19.019228 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.019240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.019352 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.019591 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.019593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.019776 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.019868 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.019873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.020921 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.020924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.021793 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.021802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.022340 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.022344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.022431 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.024380 (+ 1949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.024391 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.024730 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.026919 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.026924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.026927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.027132 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.027423 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.027426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.027428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.027516 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.027900 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.027903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.028003 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.028107 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.028261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:19.048429 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45867
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:19.160774 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:19.161010 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:19.161057 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:19.165148 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:19.165210 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:19.165251 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:38:19.165264 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:19.165382 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:19.171893 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:19.173166 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:19.174460 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:19.184448 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:19.184441 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:19.184448 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:19.184805 2333 server_base.cc:1034] running on GCE node
I20260925 16:38:19.185335 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:19.185984 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:19.187170 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790354299187146 us; error 33 us; skew 500 ppm
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:19.195767 2333 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:19.196787 2333 webserver.cc:460] Webserver started at http://127.0.97.194:40501/ using document root <none> and password file <none>
I20260925 16:38:19.197458 2333 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:19.197528 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:19.197753 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:19.199906 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "6115b4e9f93b46499d54ac4f56f5861b"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "95fa04dd7dfba62680b7787e1787f74d"
server_key_iv: "1f574ddf147ecc00353e06598807e241"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.200477 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "6115b4e9f93b46499d54ac4f56f5861b"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "95fa04dd7dfba62680b7787e1787f74d"
server_key_iv: "1f574ddf147ecc00353e06598807e241"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.204320 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:38:19.206933 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:19.208236 2333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:19.208357 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "6115b4e9f93b46499d54ac4f56f5861b"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "95fa04dd7dfba62680b7787e1787f74d"
server_key_iv: "1f574ddf147ecc00353e06598807e241"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.208463 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:19.233567 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:19.236853 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:19.237056 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:19.238607 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:19.238688 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:19.238760 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:19.238812 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:19.249841 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:46009
I20260925 16:38:19.249862 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:46009 every 8 connection(s)
I20260925 16:38:19.250787 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:19.252844 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.239491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:57211 (local address 127.0.97.254:45867)
0925 16:38:19.239622 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.239626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.240699 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 16:38:19.241878 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.241882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.241947 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.242040 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.243372 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.244003 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.245408 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.245826 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.245957 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.248720 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.248724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.248727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.248765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.250524 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.251124 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.251129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.251132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.251256 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.251684 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.251696 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.251698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.251901 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.252019 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.252521 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.252652 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:19.253685 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.240034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.194:57211)
0925 16:38:19.240534 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.240579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.241602 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.242285 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.242292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.242646 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.243197 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.243244 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.244198 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.244201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.245222 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.245233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.245994 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.246000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.246256 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.246946 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.246969 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.248524 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.250648 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.250655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.250669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.251016 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.251376 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.251379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.251381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.251555 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.252013 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.252020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.252276 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.252473 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.252790 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":146,"threads_started":1}
I20260925 16:38:19.254834 2464 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45867
I20260925 16:38:19.254992 2464 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:19.255446 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2333
I20260925 16:38:19.255553 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:19.255656 2464 heartbeater.cc:511] Master 127.0.97.254:45867 requested a full tablet report, sending...
I20260925 16:38:19.255857 391 external_mini_cluster.cc:1428] Setting key bfd02ef757d18c0caa9d52543daddd67
I20260925 16:38:19.256699 2120 ts_manager.cc:194] Registered new tserver with Master: 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009)
I20260925 16:38:19.257473 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:57211
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:19.265328 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.257298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:42525 (local address 127.0.97.254:45867)
0925 16:38:19.257420 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.257424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.257950 (+ 526us) server_negotiation.cc:407] Connection header received
0925 16:38:19.258117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.258124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.258187 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.258283 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.258762 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.259536 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.260426 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.260667 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.260792 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.261975 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.261979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.261982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.262015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.263539 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.264007 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.264012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.264014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.264097 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.264405 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.264409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.264411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.264626 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.264752 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.265025 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.265142 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:19.265344 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.257181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.194:42525)
0925 16:38:19.257860 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.257875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.257990 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.258319 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.258322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.258527 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.258636 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.258643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.259669 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.259673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.260299 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.260307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.260793 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.260797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.260897 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.261482 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.261494 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.261824 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.263679 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.263683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.263685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.263902 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.264206 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.264210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.264212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.264302 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.264739 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.264742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.264846 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.264959 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.265108 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:19.284000 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45867
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:19.394184 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:19.394423 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:19.394469 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:19.398314 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:19.398358 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:19.398394 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:38:19.398408 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:19.398445 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:19.403402 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:19.404292 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:19.405203 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:19.414561 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:19.414556 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:19.414660 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:19.414921 2473 server_base.cc:1034] running on GCE node
I20260925 16:38:19.415307 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:19.415858 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:19.417012 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790354299416992 us; error 28 us; skew 500 ppm
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:19.419762 2473 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:19.420579 2473 webserver.cc:460] Webserver started at http://127.0.97.195:43873/ using document root <none> and password file <none>
I20260925 16:38:19.421113 2473 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:19.421214 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:19.421459 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:19.423408 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "675292cae7394b0fb005a935f0c0e932"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "7c85125a94cfb91b0cb22453a306b4e9"
server_key_iv: "136b948a4d4390b0f872df8147b2548c"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.423944 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "675292cae7394b0fb005a935f0c0e932"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "7c85125a94cfb91b0cb22453a306b4e9"
server_key_iv: "136b948a4d4390b0f872df8147b2548c"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.427625 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:19.429960 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:19.431278 2473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:19.431404 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "675292cae7394b0fb005a935f0c0e932"
format_stamp: "Formatted at 2026-09-25 16:38:19 on dist-test-slave-kxl8"
server_key: "7c85125a94cfb91b0cb22453a306b4e9"
server_key_iv: "136b948a4d4390b0f872df8147b2548c"
server_key_version: "encryptionkey@0"
I20260925 16:38:19.431510 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:19.452297 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:19.455245 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:19.455459 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:19.456954 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:19.457023 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:19.457085 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:19.457127 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:19.467904 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:39435
I20260925 16:38:19.467922 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:39435 every 8 connection(s)
I20260925 16:38:19.469080 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:19.470316 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2473
I20260925 16:38:19.470440 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:19.470711 391 external_mini_cluster.cc:1428] Setting key 56af3870bee5933126980e79892c9ec3
I20260925 16:38:19.471613 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.457838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:59461 (local address 127.0.97.254:45867)
0925 16:38:19.457963 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.457967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.459046 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 16:38:19.460243 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.460249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.460316 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.460419 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.461780 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.462684 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.463920 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.464323 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.464472 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.467381 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.467384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.467386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.467411 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.469333 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.469924 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.469928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.469929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.470018 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.470440 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.470443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.470445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.470648 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.470801 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.471332 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.471450 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:19.472363 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.458391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.195:59461)
0925 16:38:19.458886 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.458929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.459904 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.460693 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.460701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.461041 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.461608 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.461629 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.462822 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.462827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.463790 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.463799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.464515 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.464520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.464740 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.465392 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.465408 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.467214 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.469453 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.469458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.469472 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.469832 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.470142 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.470146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.470149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.470331 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.470803 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.470809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.471087 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.471281 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.471581 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260925 16:38:19.473728 2604 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45867
I20260925 16:38:19.473884 2604 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:19.474378 2604 heartbeater.cc:511] Master 127.0.97.254:45867 requested a full tablet report, sending...
I20260925 16:38:19.475315 2121 ts_manager.cc:194] Registered new tserver with Master: 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:19.475944 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:59461
I20260925 16:38:19.482515 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.475876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:38333 (local address 127.0.97.254:45867)
0925 16:38:19.475989 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.475994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.476260 (+ 266us) server_negotiation.cc:407] Connection header received
0925 16:38:19.476290 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.476292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.476324 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.476380 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.476854 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.477431 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.478231 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.478461 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.478570 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.479614 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.479617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.479620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.479642 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.480935 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.481374 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.481378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.481380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.481461 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.481746 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.481751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.481753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.481921 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.482023 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.482270 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.482370 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:19.482530 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.475822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.97.195:38333)
0925 16:38:19.476067 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.476081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.476167 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.476419 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.476422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.476620 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.476740 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.476746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.477551 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.477554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.478114 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.478121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.478581 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.478585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.478674 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.479188 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.479200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.479483 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.481043 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.481048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.481050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.481247 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.481558 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.481561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.481563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.481648 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.482013 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.482016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.482111 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.482217 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.482357 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260925 16:38:19.485054 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:19.509200 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.501415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.0.97.254:45867)
0925 16:38:19.501535 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.501538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.501551 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:19.501611 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.501614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.501662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.501766 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.502449 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.502972 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.503763 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.503964 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.504036 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.506386 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.506389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.506391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.506413 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.507816 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.508212 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.508215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.508216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.508282 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.508497 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.508499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.508500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.508672 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.508786 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.508990 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.509080 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:19.512004 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:19.515084 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:19.529556 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.523319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:46009 (local address 127.0.0.1:47696)
0925 16:38:19.524283 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.524332 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.524477 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.525239 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.525242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.525278 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.525576 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.525583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.526574 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.526577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.528045 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.528054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.529132 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.529146 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.529242 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.529262 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.529345 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.529404 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":120,"spinlock_wait_cycles":5504,"thread_start_us":75,"threads_started":1}
I20260925 16:38:19.529587 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.523685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.97.195:39435)
0925 16:38:19.524728 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.524744 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.524771 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:38:19.524849 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.524855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.525043 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.525230 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.525689 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.526470 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.528021 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.528627 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.528749 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.528939 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.529370 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.529464 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":88,"threads_started":1}
I20260925 16:38:19.529623 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.523443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.97.194:46009)
0925 16:38:19.523900 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.523914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.524421 (+ 507us) server_negotiation.cc:407] Connection header received
0925 16:38:19.524677 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.524684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.524919 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.525050 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.525750 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.526456 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.528166 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.529007 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.529129 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.529334 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.529386 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.529477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":113,"threads_started":1}
I20260925 16:38:19.530215 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.523519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39435 (local address 127.0.0.1:42642)
0925 16:38:19.524336 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.524349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.524465 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.525234 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.525240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.525328 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.525556 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.525562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.526607 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.526610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.527899 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.527907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.528883 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.528886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.529013 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.529057 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.529183 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.530085 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":151,"spinlock_wait_cycles":1792,"thread_start_us":97,"threads_started":1}
I20260925 16:38:19.531080 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.524244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.193:43105)
0925 16:38:19.524594 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.524601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.524837 (+ 236us) server_negotiation.cc:407] Connection header received
0925 16:38:19.524945 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.524950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.525119 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.525261 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.526639 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.527776 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.529174 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.530026 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.530270 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.530409 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.530918 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.530991 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":72,"threads_started":1}
I20260925 16:38:19.532605 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e (DEFAULT_TABLE table=test-table [id=47a0feca4a2846a98879791eca7b0cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:19.530840 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.524067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.0.1:50032)
0925 16:38:19.524724 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.524740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.524841 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.525341 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.525344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.525357 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.525553 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.525562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.527892 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.527896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.529041 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.529053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.530465 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.530468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.530564 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.530589 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.530657 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.530739 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":58,"thread_start_us":98,"threads_started":1}
I20260925 16:38:19.533696 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07911e7bdf5d41758ea1fb86cc91f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.535861 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e (DEFAULT_TABLE table=test-table [id=47a0feca4a2846a98879791eca7b0cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:19.536231 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e (DEFAULT_TABLE table=test-table [id=47a0feca4a2846a98879791eca7b0cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:19.536825 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07911e7bdf5d41758ea1fb86cc91f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.537261 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07911e7bdf5d41758ea1fb86cc91f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.541736 2627 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Bootstrap starting.
I20260925 16:38:19.543006 2628 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Bootstrap starting.
I20260925 16:38:19.544088 2627 tablet_bootstrap.cc:654] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.544492 2629 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Bootstrap starting.
I20260925 16:38:19.545022 2628 tablet_bootstrap.cc:654] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.545168 2627 log.cc:824] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:19.545816 2628 log.cc:824] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:19.546504 2629 tablet_bootstrap.cc:654] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.547127 2629 log.cc:824] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:19.547756 2628 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: No bootstrap required, opened a new log
I20260925 16:38:19.547756 2627 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: No bootstrap required, opened a new log
I20260925 16:38:19.547940 2628 ts_tablet_manager.cc:1397] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:38:19.547940 2627 ts_tablet_manager.cc:1397] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:38:19.548760 2629 tablet_bootstrap.cc:492] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: No bootstrap required, opened a new log
I20260925 16:38:19.548951 2629 ts_tablet_manager.cc:1397] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 16:38:19.551847 2629 raft_consensus.cc:348] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.552106 2629 raft_consensus.cc:374] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.551939 2627 raft_consensus.cc:348] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.552160 2629 raft_consensus.cc:729] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675292cae7394b0fb005a935f0c0e932, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.552175 2627 raft_consensus.cc:374] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.552237 2627 raft_consensus.cc:729] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.552352 2628 raft_consensus.cc:348] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.552574 2628 raft_consensus.cc:374] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.552634 2628 raft_consensus.cc:729] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.552615 2629 consensus_queue.cc:260] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.552668 2627 consensus_queue.cc:260] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.553042 2628 consensus_queue.cc:260] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.553360 2604 heartbeater.cc:503] Master 127.0.97.254:45867 was elected leader, sending a full tablet report...
I20260925 16:38:19.553632 2627 ts_tablet_manager.cc:1428] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:38:19.553632 2629 ts_tablet_manager.cc:1428] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 16:38:19.553829 2628 ts_tablet_manager.cc:1428] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260925 16:38:19.553937 2464 heartbeater.cc:503] Master 127.0.97.254:45867 was elected leader, sending a full tablet report...
I20260925 16:38:19.554116 2324 heartbeater.cc:503] Master 127.0.97.254:45867 was elected leader, sending a full tablet report...
W20260925 16:38:19.720275 2605 tablet.cc:2369] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:19.749470 2634 raft_consensus.cc:482] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:19.749683 2634 raft_consensus.cc:504] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.750795 2634 leader_election.cc:288] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435), 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105)
W20260925 16:38:19.751938 2465 tablet.cc:2369] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:19.754956 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.751091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39435 (local address 127.0.97.194:37465)
0925 16:38:19.751216 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.751233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.751339 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.751653 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.751656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.751675 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.751804 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.751815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.752730 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.752734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.753700 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.753709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.754575 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.754578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.754648 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.754667 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.754739 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.754793 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 16:38:19.754971 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.751206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37465 (local address 127.0.97.195:39435)
0925 16:38:19.751362 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.751368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.751385 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:38:19.751447 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.751450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.751505 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.751619 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.751908 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.752610 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.753839 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.754468 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.754590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.754640 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.754782 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.754848 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:19.755785 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07911e7bdf5d41758ea1fb86cc91f35e" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932" is_pre_election: true
I20260925 16:38:19.756161 2558 raft_consensus.cc:2454] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:19.756299 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.751385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.97.194:49879)
0925 16:38:19.751730 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.751743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.751857 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.752188 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.752191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.752207 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.752333 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.752338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.753206 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.753208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.755006 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.755018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.755881 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.755884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.755990 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.756011 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.756081 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.756166 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":92,"threads_started":1}
I20260925 16:38:19.756310 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.751543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:49879 (local address 127.0.97.193:43105)
0925 16:38:19.751759 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.751763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.751828 (+ 65us) server_negotiation.cc:407] Connection header received
0925 16:38:19.752024 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.752029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.752082 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.752161 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.752443 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.753090 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.755154 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.755682 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.755757 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.755790 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.756112 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.756193 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260925 16:38:19.756700 2352 leader_election.cc:302] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115b4e9f93b46499d54ac4f56f5861b, 675292cae7394b0fb005a935f0c0e932; no voters:
I20260925 16:38:19.757014 2634 raft_consensus.cc:2781] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:19.756974 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07911e7bdf5d41758ea1fb86cc91f35e" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739" is_pre_election: true
I20260925 16:38:19.757112 2634 raft_consensus.cc:482] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:19.757159 2634 raft_consensus.cc:3037] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.757308 2278 raft_consensus.cc:2454] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:19.758296 2634 raft_consensus.cc:504] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
I20260925 16:38:19.758586 2634 leader_election.cc:288] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Requested vote from peers 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435), 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105)
I20260925 16:38:19.758903 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07911e7bdf5d41758ea1fb86cc91f35e" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932"
I20260925 16:38:19.758984 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07911e7bdf5d41758ea1fb86cc91f35e" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739"
I20260925 16:38:19.759044 2558 raft_consensus.cc:3037] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.759105 2278 raft_consensus.cc:3037] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.760155 2278 raft_consensus.cc:2454] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:19.760479 2352 leader_election.cc:302] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:19.760658 2634 raft_consensus.cc:2781] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:19.760676 2558 raft_consensus.cc:2454] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:19.760883 2634 raft_consensus.cc:686] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Becoming Leader. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Running, Role: LEADER
I20260925 16:38:19.761219 2634 consensus_queue.cc:237] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } }
W20260925 16:38:19.763305 2325 tablet.cc:2369] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:19.764506 2119 catalog_manager.cc:5495] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194). New cstate: current_term: 1 leader_uuid: "6115b4e9f93b46499d54ac4f56f5861b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:38:19 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:19.851203 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.843095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.0.97.254:45867)
0925 16:38:19.843259 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.843264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.843437 (+ 173us) server_negotiation.cc:407] Connection header received
0925 16:38:19.843546 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.843549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.843590 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.843664 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:19.844336 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.844852 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.845634 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.845848 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.845934 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.848152 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:19.848155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:19.848157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:19.848177 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:19.849701 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.850099 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.850101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.850102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.850176 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:19.850446 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:19.850449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:19.850450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:19.850620 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:19.850711 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.850964 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.851078 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":82}
I20260925 16:38:19.851205 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.843036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45867 (local address 127.0.0.1:33064)
0925 16:38:19.843341 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.843354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.843462 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.843705 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.843708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.844105 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:19.844197 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.844203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.844945 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.844947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.845512 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.845519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.845951 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.845954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.846012 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.846550 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:19.846561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:19.847942 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:19.849789 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.849796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.849801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.850014 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.850265 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:19.850267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:19.850268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:19.850344 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:19.850717 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:19.850723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:19.850821 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.850910 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.851022 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":131,"threads_started":1}
I20260925 16:38:19.855177 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:33064:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:38:19.855938 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:19.860635 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1c87c9e69c432fbf7cf83664b8efc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c000ec654f284711a5992941930fae6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:19.860630 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1c87c9e69c432fbf7cf83664b8efc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c000ec654f284711a5992941930fae6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:19.860888 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1c87c9e69c432fbf7cf83664b8efc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c000ec654f284711a5992941930fae6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:19.861418 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1c87c9e69c432fbf7cf83664b8efc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.861418 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1c87c9e69c432fbf7cf83664b8efc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.861584 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1c87c9e69c432fbf7cf83664b8efc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:19.864225 2627 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: Bootstrap starting.
I20260925 16:38:19.864239 2628 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: Bootstrap starting.
I20260925 16:38:19.865218 2628 tablet_bootstrap.cc:654] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.865218 2627 tablet_bootstrap.cc:654] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.865944 2629 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: Bootstrap starting.
I20260925 16:38:19.866619 2627 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: No bootstrap required, opened a new log
I20260925 16:38:19.866705 2627 ts_tablet_manager.cc:1397] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:19.866915 2629 tablet_bootstrap.cc:654] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:19.867200 2627 raft_consensus.cc:348] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.867326 2627 raft_consensus.cc:374] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.867362 2627 raft_consensus.cc:729] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.867533 2627 consensus_queue.cc:260] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.867836 2627 ts_tablet_manager.cc:1428] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:19.867801 2634 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.867918 2634 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.868609 2629 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: No bootstrap required, opened a new log
I20260925 16:38:19.868647 2628 tablet_bootstrap.cc:492] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: No bootstrap required, opened a new log
I20260925 16:38:19.868711 2629 ts_tablet_manager.cc:1397] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:19.868741 2628 ts_tablet_manager.cc:1397] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 16:38:19.869153 2628 raft_consensus.cc:348] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.869271 2628 raft_consensus.cc:374] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.869166 2629 raft_consensus.cc:348] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.869335 2628 raft_consensus.cc:729] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.869356 2629 raft_consensus.cc:374] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:19.869392 2629 raft_consensus.cc:729] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675292cae7394b0fb005a935f0c0e932, State: Initialized, Role: FOLLOWER
I20260925 16:38:19.869481 2628 consensus_queue.cc:260] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.869535 2629 consensus_queue.cc:260] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.869755 2628 ts_tablet_manager.cc:1428] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:19.869791 2629 ts_tablet_manager.cc:1428] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:19.869984 2635 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.870014 2633 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.870110 2635 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.870148 2633 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.911021 2635 raft_consensus.cc:482] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:19.911182 2635 raft_consensus.cc:504] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.912411 2635 leader_election.cc:288] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:19.916334 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.912751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:40659 (local address 127.0.97.194:46009)
0925 16:38:19.912891 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.912895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.913115 (+ 220us) server_negotiation.cc:407] Connection header received
0925 16:38:19.913319 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.913322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.913370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.913462 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.913744 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.914256 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.915342 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.915758 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.915898 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.915955 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.916136 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.916202 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:38:19.916360 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.912606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:46009 (local address 127.0.97.193:40659)
0925 16:38:19.913004 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.913018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.913186 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.913474 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.913476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.913492 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.913634 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.913640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.914367 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.914372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.915216 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.915225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.915898 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.915901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.916005 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.916034 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.916122 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.916202 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":27,"thread_start_us":153,"threads_started":1}
I20260925 16:38:19.916620 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.912882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:38515 (local address 127.0.97.195:39435)
0925 16:38:19.912998 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.913002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.913184 (+ 182us) server_negotiation.cc:407] Connection header received
0925 16:38:19.913388 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.913392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.913431 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.913518 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:19.914205 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.914674 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.915668 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.916044 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.916170 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.916218 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.916391 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.916501 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 16:38:19.916889 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1c87c9e69c432fbf7cf83664b8efc0" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115b4e9f93b46499d54ac4f56f5861b" is_pre_election: true
I20260925 16:38:19.917017 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.912798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39435 (local address 127.0.97.193:38515)
0925 16:38:19.913076 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.913087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.913251 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.913526 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.913529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.913544 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:19.914089 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.914096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.914791 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.914797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.915549 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.915557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.916158 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.916162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.916275 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.916291 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.916376 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.916885 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":21,"thread_start_us":107,"threads_started":1}
I20260925 16:38:19.917120 2415 raft_consensus.cc:2454] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 0.
I20260925 16:38:19.917486 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1c87c9e69c432fbf7cf83664b8efc0" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932" is_pre_election: true
I20260925 16:38:19.917623 2558 raft_consensus.cc:2454] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 0.
I20260925 16:38:19.917618 2211 leader_election.cc:302] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:19.917910 2635 raft_consensus.cc:2781] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:19.917974 2635 raft_consensus.cc:482] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:19.918013 2635 raft_consensus.cc:3037] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.919008 2635 raft_consensus.cc:504] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.919341 2635 leader_election.cc:288] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 election: Requested vote from peers 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:19.919658 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1c87c9e69c432fbf7cf83664b8efc0" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115b4e9f93b46499d54ac4f56f5861b"
I20260925 16:38:19.919766 2415 raft_consensus.cc:3037] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.919734 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1c87c9e69c432fbf7cf83664b8efc0" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932"
I20260925 16:38:19.919873 2558 raft_consensus.cc:3037] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:19.920624 2415 raft_consensus.cc:2454] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 1.
I20260925 16:38:19.920804 2558 raft_consensus.cc:2454] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 1.
I20260925 16:38:19.920877 2211 leader_election.cc:302] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:19.921064 2635 raft_consensus.cc:2781] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:19.921326 2635 raft_consensus.cc:686] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [term 1 LEADER]: Becoming Leader. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Running, Role: LEADER
I20260925 16:38:19.921697 2635 consensus_queue.cc:237] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:19.923995 2650 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: TxnStatusTablet state changed. Reason: New leader 328f1a52705e4f5181f59ef3115e5739. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.924104 2650 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:19.924383 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:38:19.924585 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:38:19.925148 2121 catalog_manager.cc:5495] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:19.948191 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.944136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.97.193:43105)
0925 16:38:19.944268 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.944271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.944309 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:38:19.944542 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.944544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.944599 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.944685 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:19.944951 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.945512 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.946724 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.946938 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.947026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.947350 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:19.947812 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:19.947900 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.948029 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.948082 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:19.948222 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.944015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.0.1:50042)
0925 16:38:19.944248 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.944264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.944375 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.944723 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.944726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.944736 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:19.944844 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.944852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.945607 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.945611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.946586 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.946596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.947050 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.947054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.947153 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.947205 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:19.947905 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:19.947918 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.947990 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.948081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 16:38:19.953488 2415 raft_consensus.cc:1261] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Refusing update from remote peer 328f1a52705e4f5181f59ef3115e5739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:19.953542 2558 raft_consensus.cc:1261] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Refusing update from remote peer 328f1a52705e4f5181f59ef3115e5739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:19.954156 2650 consensus_queue.cc:1035] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:19.954391 2635 consensus_queue.cc:1035] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:19.957397 2634 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: TxnStatusTablet state changed. Reason: New leader 328f1a52705e4f5181f59ef3115e5739. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.957516 2634 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.958168 2633 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: TxnStatusTablet state changed. Reason: New leader 328f1a52705e4f5181f59ef3115e5739. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.958292 2633 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.960564 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333291212602777600 to 7333291212481994752. Current Snapshot: MvccSnapshot[applied={T|T < 7333291212602777600}]
I20260925 16:38:19.960796 2650 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.960896 2650 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:19.961125 2634 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.961228 2634 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 6115b4e9f93b46499d54ac4f56f5861b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.961613 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333291212602777600 to 7333291212481994752. Current Snapshot: MvccSnapshot[applied={T|T < 7333291212602777600}]
I20260925 16:38:19.967617 2635 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.967761 2635 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 328f1a52705e4f5181f59ef3115e5739: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:19.969054 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333291212602777600 to 7333291212481994752. Current Snapshot: MvccSnapshot[applied={T|T < 7333291212602777600}]
I20260925 16:38:19.969261 2633 tablet_replica.cc:439] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } } }
I20260925 16:38:19.969390 2633 tablet_replica.cc:442] T 1b1c87c9e69c432fbf7cf83664b8efc0 P 675292cae7394b0fb005a935f0c0e932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:19.972249 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.967272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.0.97.194:46009)
0925 16:38:19.967904 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.967914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.967928 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:19.967998 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.968001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.968058 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.968156 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:19.968593 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.969405 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.970588 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.970877 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.970978 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.971388 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:19.971882 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:19.972008 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.972068 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.972122 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
I20260925 16:38:19.983402 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.979481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.97.194:55861)
0925 16:38:19.979800 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:19.979814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:19.979898 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:19.980139 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:19.980141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:19.980146 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:19.980246 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.980251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.981255 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.981261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.981969 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:19.981978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.982558 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.982562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:19.982675 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.982782 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:19.983127 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:19.983133 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:38:19.983195 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 16:38:19.983260 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260925 16:38:19.983402 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:19.979586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:55861 (local address 127.0.97.193:43105)
0925 16:38:19.979717 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:19.979722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:19.979880 (+ 158us) server_negotiation.cc:407] Connection header received
0925 16:38:19.979981 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:19.979984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:19.980046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:19.980126 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:19.980336 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.981129 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.982084 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:19.982425 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:19.982528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:19.982949 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:19.983028 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:19.983100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:19.983222 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 16:38:19.983273 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:19.990485 2558 raft_consensus.cc:1261] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:19.990643 2278 raft_consensus.cc:1261] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:19.991021 2637 consensus_queue.cc:1035] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:19.991180 2634 consensus_queue.cc:1035] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:20.018743 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:20.014450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:58193 (local address 127.0.97.194:46009)
0925 16:38:20.014569 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:20.014573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:20.014702 (+ 129us) server_negotiation.cc:407] Connection header received
0925 16:38:20.014813 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:20.014817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:20.014869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:20.014952 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:20.015236 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.016136 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.017496 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.017871 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.017979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:20.018291 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:20.018374 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:20.018448 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:20.018557 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 16:38:20.018608 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:20.019091 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:20.014102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:46009 (local address 127.0.97.193:58193)
0925 16:38:20.014608 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:20.014623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:20.014728 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:20.014972 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:20.014975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:20.014994 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:20.015120 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:20.015126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.016273 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.016277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:20.017366 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:20.017377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.017993 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.017997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:20.018096 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:20.018152 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:20.018467 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:20.018474 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:38:20.018549 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:38:20.018629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":119,"threads_started":1}
I20260925 16:38:20.070771 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:20.070998 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:20.074080 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Sending DeleteTablet for 3 replicas of tablet 07911e7bdf5d41758ea1fb86cc91f35e
I20260925 16:38:20.074842 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:20 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:47696
I20260925 16:38:20.075443 2675 tablet_replica.cc:330] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: stopping tablet replica
I20260925 16:38:20.075598 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:20 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42642
I20260925 16:38:20.075886 2675 raft_consensus.cc:2229] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:20.076323 2675 raft_consensus.cc:2258] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:20.076864 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 07911e7bdf5d41758ea1fb86cc91f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:20 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 16:38:20.077538 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:20.077780 2675 ts_tablet_manager.cc:1905] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:38:20.078102 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:20.079041 2676 tablet_replica.cc:330] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: stopping tablet replica
I20260925 16:38:20.079396 2676 raft_consensus.cc:2229] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:20.079651 2676 raft_consensus.cc:2258] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:20.079994 2677 tablet_replica.cc:330] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: stopping tablet replica
I20260925 16:38:20.080288 2677 raft_consensus.cc:2229] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:20.080487 2677 raft_consensus.cc:2258] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:20.082499 2676 ts_tablet_manager.cc:1905] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:20.083632 2397 tablet_service.cc:1467] Processing CreateTablet for tablet aeedbd9e5cf84fa7b396095789094620 (DEFAULT_TABLE table=test-table [id=aea3de966ae348498e3bbd4e04369d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:20.083724 2256 tablet_service.cc:1467] Processing CreateTablet for tablet aeedbd9e5cf84fa7b396095789094620 (DEFAULT_TABLE table=test-table [id=aea3de966ae348498e3bbd4e04369d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:20.083874 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeedbd9e5cf84fa7b396095789094620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:20.084002 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeedbd9e5cf84fa7b396095789094620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:20.084524 2677 ts_tablet_manager.cc:1905] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:20.086212 2627 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Bootstrap starting.
I20260925 16:38:20.086609 2628 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Bootstrap starting.
I20260925 16:38:20.086858 2538 tablet_service.cc:1467] Processing CreateTablet for tablet aeedbd9e5cf84fa7b396095789094620 (DEFAULT_TABLE table=test-table [id=aea3de966ae348498e3bbd4e04369d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:20.087119 2627 tablet_bootstrap.cc:654] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:20.087158 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeedbd9e5cf84fa7b396095789094620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:20.087618 2628 tablet_bootstrap.cc:654] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:20.089851 2676 ts_tablet_manager.cc:1918] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:20.089891 2629 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Bootstrap starting.
I20260925 16:38:20.089959 2676 log.cc:1196] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/07911e7bdf5d41758ea1fb86cc91f35e
I20260925 16:38:20.090327 2676 ts_tablet_manager.cc:1939] T 07911e7bdf5d41758ea1fb86cc91f35e P 675292cae7394b0fb005a935f0c0e932: Deleting consensus metadata
I20260925 16:38:20.090742 2629 tablet_bootstrap.cc:654] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:20.091662 2108 catalog_manager.cc:4841] TS 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435): tablet 07911e7bdf5d41758ea1fb86cc91f35e (table test-table [id=47a0feca4a2846a98879791eca7b0cb0]) successfully deleted
I20260925 16:38:20.092183 2675 ts_tablet_manager.cc:1918] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:20.092339 2675 log.cc:1196] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/07911e7bdf5d41758ea1fb86cc91f35e
I20260925 16:38:20.092716 2675 ts_tablet_manager.cc:1939] T 07911e7bdf5d41758ea1fb86cc91f35e P 6115b4e9f93b46499d54ac4f56f5861b: Deleting consensus metadata
I20260925 16:38:20.093097 2677 ts_tablet_manager.cc:1918] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:20.093219 2677 log.cc:1196] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/07911e7bdf5d41758ea1fb86cc91f35e
I20260925 16:38:20.093624 2677 ts_tablet_manager.cc:1939] T 07911e7bdf5d41758ea1fb86cc91f35e P 328f1a52705e4f5181f59ef3115e5739: Deleting consensus metadata
I20260925 16:38:20.094915 2108 catalog_manager.cc:4841] TS 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105): tablet 07911e7bdf5d41758ea1fb86cc91f35e (table test-table [id=47a0feca4a2846a98879791eca7b0cb0]) successfully deleted
I20260925 16:38:20.095203 2107 catalog_manager.cc:4841] TS 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009): tablet 07911e7bdf5d41758ea1fb86cc91f35e (table test-table [id=47a0feca4a2846a98879791eca7b0cb0]) successfully deleted
I20260925 16:38:20.095832 2627 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: No bootstrap required, opened a new log
I20260925 16:38:20.095914 2627 ts_tablet_manager.cc:1397] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 16:38:20.095983 2628 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: No bootstrap required, opened a new log
I20260925 16:38:20.096069 2628 ts_tablet_manager.cc:1397] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 16:38:20.096370 2627 raft_consensus.cc:348] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.096483 2627 raft_consensus.cc:374] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:20.096514 2627 raft_consensus.cc:729] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Initialized, Role: FOLLOWER
I20260925 16:38:20.096534 2628 raft_consensus.cc:348] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.096661 2628 raft_consensus.cc:374] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:20.096628 2627 consensus_queue.cc:260] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.096691 2628 raft_consensus.cc:729] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Initialized, Role: FOLLOWER
I20260925 16:38:20.096812 2628 consensus_queue.cc:260] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.096877 2627 ts_tablet_manager.cc:1428] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:20.097080 2628 ts_tablet_manager.cc:1428] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 16:38:20.100536 2635 raft_consensus.cc:482] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:20.100646 2635 raft_consensus.cc:504] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.101028 2635 leader_election.cc:288] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:20.101330 2629 tablet_bootstrap.cc:492] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: No bootstrap required, opened a new log
I20260925 16:38:20.101419 2629 ts_tablet_manager.cc:1397] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 16:38:20.101588 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeedbd9e5cf84fa7b396095789094620" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115b4e9f93b46499d54ac4f56f5861b" is_pre_election: true
I20260925 16:38:20.101706 2415 raft_consensus.cc:2454] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 0.
I20260925 16:38:20.101701 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeedbd9e5cf84fa7b396095789094620" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932" is_pre_election: true
I20260925 16:38:20.101891 2629 raft_consensus.cc:348] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.101998 2629 raft_consensus.cc:374] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:20.102002 2211 leader_election.cc:302] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:20.102035 2629 raft_consensus.cc:729] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675292cae7394b0fb005a935f0c0e932, State: Initialized, Role: FOLLOWER
I20260925 16:38:20.102160 2635 raft_consensus.cc:2781] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:20.102205 2635 raft_consensus.cc:482] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:20.102176 2629 consensus_queue.cc:260] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.102241 2635 raft_consensus.cc:3037] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Advancing to term 1
W20260925 16:38:20.102299 2212 leader_election.cc:341] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435): Illegal state: must be running to vote when last-logged opid is not known
I20260925 16:38:20.102478 2629 ts_tablet_manager.cc:1428] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:20.103394 2635 raft_consensus.cc:504] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.103788 2635 leader_election.cc:288] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 election: Requested vote from peers 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:20.104290 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeedbd9e5cf84fa7b396095789094620" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115b4e9f93b46499d54ac4f56f5861b"
I20260925 16:38:20.104282 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeedbd9e5cf84fa7b396095789094620" candidate_uuid: "328f1a52705e4f5181f59ef3115e5739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932"
I20260925 16:38:20.104388 2558 raft_consensus.cc:3037] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:20.104388 2415 raft_consensus.cc:3037] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:20.105468 2415 raft_consensus.cc:2454] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 1.
I20260925 16:38:20.105468 2558 raft_consensus.cc:2454] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328f1a52705e4f5181f59ef3115e5739 in term 1.
I20260925 16:38:20.105746 2211 leader_election.cc:302] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:20.105896 2635 raft_consensus.cc:2781] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:20.105973 2635 raft_consensus.cc:686] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 1 LEADER]: Becoming Leader. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Running, Role: LEADER
I20260925 16:38:20.106113 2635 consensus_queue.cc:237] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:20.107623 2121 catalog_manager.cc:5495] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "328f1a52705e4f5181f59ef3115e5739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:20.128294 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:20.124367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.0.97.193:43105)
0925 16:38:20.124482 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:20.124486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:20.124500 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:20.124570 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:20.124573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:20.124622 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:20.124706 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:20.124951 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.125859 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.127060 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.127405 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.127546 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:20.127790 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:20.127883 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:20.127963 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:20.128087 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 16:38:20.128144 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:38:20.138414 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:20.134932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.97.193:41471)
0925 16:38:20.135051 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:20.135065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:20.135174 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:20.135397 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:20.135400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:20.135407 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:20.135528 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:20.135533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.136534 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.136536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:20.137133 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:20.137141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.137743 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.137746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:20.137837 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:20.137878 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:20.138150 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:20.138154 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:38:20.138214 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:38:20.138291 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":2}
I20260925 16:38:20.138441 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:20.135054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:41471 (local address 127.0.97.193:43105)
0925 16:38:20.135172 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:20.135176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:20.135190 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:20.135263 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:20.135266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:20.135306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:20.135370 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:20.135660 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.136429 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.137257 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:20.137630 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:20.137719 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:20.137982 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:20.138053 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:20.138129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:20.138228 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 16:38:20.138304 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":61}
I20260925 16:38:20.143776 2415 raft_consensus.cc:1261] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Refusing update from remote peer 328f1a52705e4f5181f59ef3115e5739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:20.143990 2558 raft_consensus.cc:1261] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Refusing update from remote peer 328f1a52705e4f5181f59ef3115e5739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:20.144404 2650 consensus_queue.cc:1035] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:20.144637 2635 consensus_queue.cc:1035] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:38:20.158622 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:38:21.725791 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:21.725939 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:21.728391 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Sending DeleteTablet for 3 replicas of tablet aeedbd9e5cf84fa7b396095789094620
I20260925 16:38:21.728971 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet aeedbd9e5cf84fa7b396095789094620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 16:38:21.729115 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet aeedbd9e5cf84fa7b396095789094620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:47696
I20260925 16:38:21.729127 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet aeedbd9e5cf84fa7b396095789094620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42642
I20260925 16:38:21.729455 2704 tablet_replica.cc:330] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: stopping tablet replica
I20260925 16:38:21.729597 2704 raft_consensus.cc:2229] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:21.729929 2704 raft_consensus.cc:2258] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:21.730551 2704 ts_tablet_manager.cc:1905] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:21.732494 2705 tablet_replica.cc:330] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: stopping tablet replica
I20260925 16:38:21.732664 2705 raft_consensus.cc:2229] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:21.732813 2705 raft_consensus.cc:2258] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:21.733067 2706 tablet_replica.cc:330] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: stopping tablet replica
I20260925 16:38:21.733243 2706 raft_consensus.cc:2229] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:21.733403 2706 raft_consensus.cc:2258] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:21.733999 2705 ts_tablet_manager.cc:1905] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:21.734160 2704 ts_tablet_manager.cc:1918] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:21.734256 2704 log.cc:1196] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/aeedbd9e5cf84fa7b396095789094620
I20260925 16:38:21.734992 2704 ts_tablet_manager.cc:1939] T aeedbd9e5cf84fa7b396095789094620 P 328f1a52705e4f5181f59ef3115e5739: Deleting consensus metadata
I20260925 16:38:21.735771 2108 catalog_manager.cc:4841] TS 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105): tablet aeedbd9e5cf84fa7b396095789094620 (table test-table [id=aea3de966ae348498e3bbd4e04369d8e]) successfully deleted
I20260925 16:38:21.736559 2705 ts_tablet_manager.cc:1918] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:21.736619 2705 log.cc:1196] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/aeedbd9e5cf84fa7b396095789094620
I20260925 16:38:21.736922 2705 ts_tablet_manager.cc:1939] T aeedbd9e5cf84fa7b396095789094620 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting consensus metadata
I20260925 16:38:21.737674 2107 catalog_manager.cc:4841] TS 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009): tablet aeedbd9e5cf84fa7b396095789094620 (table test-table [id=aea3de966ae348498e3bbd4e04369d8e]) successfully deleted
I20260925 16:38:21.737800 2706 ts_tablet_manager.cc:1905] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:21.739877 2706 ts_tablet_manager.cc:1918] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:21.739979 2706 log.cc:1196] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/aeedbd9e5cf84fa7b396095789094620
I20260925 16:38:21.740257 2706 ts_tablet_manager.cc:1939] T aeedbd9e5cf84fa7b396095789094620 P 675292cae7394b0fb005a935f0c0e932: Deleting consensus metadata
I20260925 16:38:21.740940 2108 catalog_manager.cc:4841] TS 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435): tablet aeedbd9e5cf84fa7b396095789094620 (table test-table [id=aea3de966ae348498e3bbd4e04369d8e]) successfully deleted
Sep 25 16:38:21 dist-test-slave-kxl8 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:38:21 dist-test-slave-kxl8 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:21.756261 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:21.748092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.97.254:45867)
0925 16:38:21.748333 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:21.748340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:21.748353 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:21.748393 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:21.748396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:21.748435 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:21.748515 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:21.749147 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:21.749752 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:21.750563 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:21.750784 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:21.750863 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:21.753073 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:21.753077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:21.753080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:21.753115 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:21.754718 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:21.755105 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:21.755108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:21.755109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:21.755216 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:21.755435 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:21.755437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:21.755438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:21.755637 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:21.755756 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:21.756006 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 16:38:21.756103 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260925 16:38:21.758759 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:21.759183 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:21.763832 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (DEFAULT_TABLE table=test-table [id=5833491a0ae84f68b0ae5ac374c30616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:21.763993 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (DEFAULT_TABLE table=test-table [id=5833491a0ae84f68b0ae5ac374c30616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:21.764024 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (DEFAULT_TABLE table=test-table [id=5833491a0ae84f68b0ae5ac374c30616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:21.764134 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e31a75dc4cd4cdf8a5b3400cb386975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:21.764262 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e31a75dc4cd4cdf8a5b3400cb386975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:21.764320 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e31a75dc4cd4cdf8a5b3400cb386975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:21.766465 2715 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Bootstrap starting.
I20260925 16:38:21.767272 2717 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Bootstrap starting.
I20260925 16:38:21.767612 2715 tablet_bootstrap.cc:654] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:21.768306 2717 tablet_bootstrap.cc:654] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:21.769003 2715 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: No bootstrap required, opened a new log
I20260925 16:38:21.769120 2715 ts_tablet_manager.cc:1397] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:21.769518 2716 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Bootstrap starting.
I20260925 16:38:21.769635 2715 raft_consensus.cc:348] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.769760 2715 raft_consensus.cc:374] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:21.769804 2715 raft_consensus.cc:729] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Initialized, Role: FOLLOWER
I20260925 16:38:21.769948 2715 consensus_queue.cc:260] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.770328 2715 ts_tablet_manager.cc:1428] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:38:21.770522 2716 tablet_bootstrap.cc:654] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:21.771472 2717 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: No bootstrap required, opened a new log
I20260925 16:38:21.771595 2717 ts_tablet_manager.cc:1397] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 16:38:21.771824 2716 tablet_bootstrap.cc:492] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: No bootstrap required, opened a new log
I20260925 16:38:21.771919 2716 ts_tablet_manager.cc:1397] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:21.772113 2717 raft_consensus.cc:348] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.772238 2717 raft_consensus.cc:374] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:21.772281 2717 raft_consensus.cc:729] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675292cae7394b0fb005a935f0c0e932, State: Initialized, Role: FOLLOWER
I20260925 16:38:21.772418 2717 consensus_queue.cc:260] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.772542 2716 raft_consensus.cc:348] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.772678 2716 raft_consensus.cc:374] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:21.772714 2716 raft_consensus.cc:729] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Initialized, Role: FOLLOWER
I20260925 16:38:21.772866 2716 consensus_queue.cc:260] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:21.773321 2716 ts_tablet_manager.cc:1428] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:21.773454 2717 ts_tablet_manager.cc:1428] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 16:38:22.076450 2722 raft_consensus.cc:482] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:22.076597 2722 raft_consensus.cc:504] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.077917 2722 leader_election.cc:288] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105), 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009)
I20260925 16:38:22.078905 2720 raft_consensus.cc:482] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:22.079023 2720 raft_consensus.cc:504] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.079468 2720 leader_election.cc:288] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:22.079865 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932" is_pre_election: true
I20260925 16:38:22.079924 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739" is_pre_election: true
I20260925 16:38:22.079988 2558 raft_consensus.cc:2454] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:22.080070 2278 raft_consensus.cc:2454] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:22.080272 2352 leader_election.cc:302] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115b4e9f93b46499d54ac4f56f5861b, 675292cae7394b0fb005a935f0c0e932; no voters:
I20260925 16:38:22.080559 2720 raft_consensus.cc:2781] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:22.080618 2720 raft_consensus.cc:482] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:22.080665 2720 raft_consensus.cc:3037] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.081938 2720 raft_consensus.cc:504] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.082261 2720 leader_election.cc:288] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Requested vote from peers 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:22.082578 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932"
I20260925 16:38:22.082613 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739"
I20260925 16:38:22.082710 2558 raft_consensus.cc:3037] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.082743 2278 raft_consensus.cc:3037] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.083029 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:22.078621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:50979 (local address 127.0.97.194:46009)
0925 16:38:22.078906 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:22.078916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:22.078932 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:38:22.078988 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:22.078993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:22.079041 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:22.079266 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:22.079568 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.080272 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.081546 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.081961 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.082149 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:22.082191 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:22.082742 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 16:38:22.082860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260925 16:38:22.083158 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:22.078269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:35443 (local address 127.0.97.193:43105)
0925 16:38:22.078570 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:22.078584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:22.078625 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:38:22.078791 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:22.078795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:22.078847 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:22.078928 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:22.079404 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.080139 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.081627 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.082244 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.082364 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:22.082413 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:22.082953 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 16:38:22.083036 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260925 16:38:22.083442 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:22.078158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:46009 (local address 127.0.97.195:50979)
0925 16:38:22.078508 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:22.078525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:22.078636 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:22.079277 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:22.079281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:22.079326 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:22.079443 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:22.079451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.080422 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.080427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:22.081431 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:22.081440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.082263 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.082267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:22.082410 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:22.082442 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 16:38:22.082510 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:38:22.082612 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"spinlock_wait_cycles":2304,"thread_start_us":165,"threads_started":1}
I20260925 16:38:22.083456 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:22.078171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.97.195:35443)
0925 16:38:22.078547 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:22.078562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:22.078681 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:22.078937 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:22.078940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:22.078955 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:22.079281 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:22.079288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.080278 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.080283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:22.081480 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:22.081491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.082468 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.082471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:22.082567 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:22.082590 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:38:22.082686 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:38:22.082770 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":155,"threads_started":1}
I20260925 16:38:22.083918 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "675292cae7394b0fb005a935f0c0e932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115b4e9f93b46499d54ac4f56f5861b" is_pre_election: true
I20260925 16:38:22.084036 2558 raft_consensus.cc:2454] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:22.084092 2417 raft_consensus.cc:2379] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675292cae7394b0fb005a935f0c0e932 in current term 1: Already voted for candidate 6115b4e9f93b46499d54ac4f56f5861b in this term.
I20260925 16:38:22.084084 2278 raft_consensus.cc:2454] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:22.084337 2352 leader_election.cc:302] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115b4e9f93b46499d54ac4f56f5861b, 675292cae7394b0fb005a935f0c0e932; no voters:
I20260925 16:38:22.084496 2720 raft_consensus.cc:2781] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:22.084530 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e31a75dc4cd4cdf8a5b3400cb386975" candidate_uuid: "675292cae7394b0fb005a935f0c0e932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739" is_pre_election: true
I20260925 16:38:22.084692 2278 raft_consensus.cc:2379] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675292cae7394b0fb005a935f0c0e932 in current term 1: Already voted for candidate 6115b4e9f93b46499d54ac4f56f5861b in this term.
I20260925 16:38:22.084707 2720 raft_consensus.cc:686] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Becoming Leader. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Running, Role: LEADER
I20260925 16:38:22.084847 2720 consensus_queue.cc:237] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.084966 2492 leader_election.cc:302] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675292cae7394b0fb005a935f0c0e932; no voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b
I20260925 16:38:22.085229 2722 raft_consensus.cc:2726] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 16:38:22.086494 2121 catalog_manager.cc:5495] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194). New cstate: current_term: 1 leader_uuid: "6115b4e9f93b46499d54ac4f56f5861b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:22.140229 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:22.137573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.97.194:46009)
0925 16:38:22.137685 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:22.137696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:22.137710 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:22.137789 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:22.137791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:22.137830 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:22.137892 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:22.138118 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.138608 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.139389 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:22.139587 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:22.139659 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:22.139887 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:22.139947 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:22.139998 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:22.140080 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:38:22.140121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:38:22.146443 2558 raft_consensus.cc:1261] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:22.146505 2278 raft_consensus.cc:1261] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:22.146922 2720 consensus_queue.cc:1035] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:22.147152 2727 consensus_queue.cc:1035] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:22.179498 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:22.179636 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:22.181396 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Sending DeleteTablet for 3 replicas of tablet 7e31a75dc4cd4cdf8a5b3400cb386975
I20260925 16:38:22.181874 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 16:38:22.181936 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:47696
I20260925 16:38:22.182075 2705 tablet_replica.cc:330] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: stopping tablet replica
I20260925 16:38:22.182075 2704 tablet_replica.cc:330] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: stopping tablet replica
I20260925 16:38:22.182072 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e31a75dc4cd4cdf8a5b3400cb386975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42642
I20260925 16:38:22.182183 2704 raft_consensus.cc:2229] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:22.182184 2705 raft_consensus.cc:2229] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:22.182257 2706 tablet_replica.cc:330] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: stopping tablet replica
I20260925 16:38:22.182289 2704 raft_consensus.cc:2258] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:22.182377 2706 raft_consensus.cc:2229] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:22.182384 2705 raft_consensus.cc:2258] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:22.182490 2706 raft_consensus.cc:2258] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:22.183075 2705 ts_tablet_manager.cc:1905] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:22.183202 2704 ts_tablet_manager.cc:1905] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:22.183274 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:22.183436 2706 ts_tablet_manager.cc:1905] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:38:22.183647 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:22.185739 2704 ts_tablet_manager.cc:1918] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:22.185738 2706 ts_tablet_manager.cc:1918] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:22.185815 2706 log.cc:1196] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/7e31a75dc4cd4cdf8a5b3400cb386975
I20260925 16:38:22.185814 2704 log.cc:1196] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/7e31a75dc4cd4cdf8a5b3400cb386975
I20260925 16:38:22.186070 2704 ts_tablet_manager.cc:1939] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 328f1a52705e4f5181f59ef3115e5739: Deleting consensus metadata
I20260925 16:38:22.186070 2706 ts_tablet_manager.cc:1939] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 675292cae7394b0fb005a935f0c0e932: Deleting consensus metadata
I20260925 16:38:22.186340 2705 ts_tablet_manager.cc:1918] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:22.186405 2705 log.cc:1196] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/7e31a75dc4cd4cdf8a5b3400cb386975
I20260925 16:38:22.186676 2705 ts_tablet_manager.cc:1939] T 7e31a75dc4cd4cdf8a5b3400cb386975 P 6115b4e9f93b46499d54ac4f56f5861b: Deleting consensus metadata
I20260925 16:38:22.186677 2108 catalog_manager.cc:4841] TS 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105): tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (table test-table [id=5833491a0ae84f68b0ae5ac374c30616]) successfully deleted
I20260925 16:38:22.187314 2107 catalog_manager.cc:4841] TS 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009): tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (table test-table [id=5833491a0ae84f68b0ae5ac374c30616]) successfully deleted
I20260925 16:38:22.187840 2108 catalog_manager.cc:4841] TS 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435): tablet 7e31a75dc4cd4cdf8a5b3400cb386975 (table test-table [id=5833491a0ae84f68b0ae5ac374c30616]) successfully deleted
I20260925 16:38:22.188472 2258 tablet_service.cc:1467] Processing CreateTablet for tablet adaee7214cf141e9a3ca9e2b1379904f (DEFAULT_TABLE table=test-table [id=d23bb4ed8ca2433f9142795472e0bd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:22.188684 2397 tablet_service.cc:1467] Processing CreateTablet for tablet adaee7214cf141e9a3ca9e2b1379904f (DEFAULT_TABLE table=test-table [id=d23bb4ed8ca2433f9142795472e0bd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:22.188684 2538 tablet_service.cc:1467] Processing CreateTablet for tablet adaee7214cf141e9a3ca9e2b1379904f (DEFAULT_TABLE table=test-table [id=d23bb4ed8ca2433f9142795472e0bd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:22.188750 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaee7214cf141e9a3ca9e2b1379904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:22.188920 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaee7214cf141e9a3ca9e2b1379904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:22.188920 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaee7214cf141e9a3ca9e2b1379904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:22.190650 2716 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Bootstrap starting.
I20260925 16:38:22.190922 2717 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Bootstrap starting.
I20260925 16:38:22.190980 2715 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Bootstrap starting.
I20260925 16:38:22.191444 2716 tablet_bootstrap.cc:654] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:22.191730 2715 tablet_bootstrap.cc:654] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:22.191807 2717 tablet_bootstrap.cc:654] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:22.192600 2716 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: No bootstrap required, opened a new log
I20260925 16:38:22.192682 2716 ts_tablet_manager.cc:1397] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:22.193004 2717 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: No bootstrap required, opened a new log
I20260925 16:38:22.193109 2717 ts_tablet_manager.cc:1397] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 16:38:22.193141 2716 raft_consensus.cc:348] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.193261 2716 raft_consensus.cc:374] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:22.193351 2716 raft_consensus.cc:729] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328f1a52705e4f5181f59ef3115e5739, State: Initialized, Role: FOLLOWER
I20260925 16:38:22.193490 2716 consensus_queue.cc:260] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.193593 2717 raft_consensus.cc:348] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.193712 2717 raft_consensus.cc:374] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:22.193744 2717 raft_consensus.cc:729] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675292cae7394b0fb005a935f0c0e932, State: Initialized, Role: FOLLOWER
I20260925 16:38:22.193797 2716 ts_tablet_manager.cc:1428] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:38:22.193861 2717 consensus_queue.cc:260] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.194113 2717 ts_tablet_manager.cc:1428] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:22.194697 2715 tablet_bootstrap.cc:492] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: No bootstrap required, opened a new log
I20260925 16:38:22.194790 2715 ts_tablet_manager.cc:1397] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:38:22.195178 2715 raft_consensus.cc:348] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.195324 2715 raft_consensus.cc:374] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:22.195367 2715 raft_consensus.cc:729] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Initialized, Role: FOLLOWER
I20260925 16:38:22.195484 2715 consensus_queue.cc:260] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.195747 2715 ts_tablet_manager.cc:1428] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:22.372836 2733 raft_consensus.cc:482] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:22.372970 2733 raft_consensus.cc:504] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.373404 2733 leader_election.cc:288] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:22.374013 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaee7214cf141e9a3ca9e2b1379904f" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739" is_pre_election: true
I20260925 16:38:22.374039 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaee7214cf141e9a3ca9e2b1379904f" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932" is_pre_election: true
I20260925 16:38:22.374181 2278 raft_consensus.cc:2454] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:22.374200 2558 raft_consensus.cc:2454] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 0.
I20260925 16:38:22.374531 2352 leader_election.cc:302] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:22.374770 2733 raft_consensus.cc:2781] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:22.374837 2733 raft_consensus.cc:482] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:22.374872 2733 raft_consensus.cc:3037] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.376034 2733 raft_consensus.cc:504] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.376333 2733 leader_election.cc:288] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Requested vote from peers 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105), 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435)
I20260925 16:38:22.376607 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaee7214cf141e9a3ca9e2b1379904f" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328f1a52705e4f5181f59ef3115e5739"
I20260925 16:38:22.376715 2278 raft_consensus.cc:3037] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.376673 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaee7214cf141e9a3ca9e2b1379904f" candidate_uuid: "6115b4e9f93b46499d54ac4f56f5861b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675292cae7394b0fb005a935f0c0e932"
I20260925 16:38:22.376894 2558 raft_consensus.cc:3037] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:22.377764 2278 raft_consensus.cc:2454] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:22.378014 2558 raft_consensus.cc:2454] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115b4e9f93b46499d54ac4f56f5861b in term 1.
I20260925 16:38:22.378041 2352 leader_election.cc:302] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328f1a52705e4f5181f59ef3115e5739, 6115b4e9f93b46499d54ac4f56f5861b; no voters:
I20260925 16:38:22.378178 2733 raft_consensus.cc:2781] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:22.378243 2733 raft_consensus.cc:686] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Becoming Leader. State: Replica: 6115b4e9f93b46499d54ac4f56f5861b, State: Running, Role: LEADER
I20260925 16:38:22.378355 2733 consensus_queue.cc:237] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } }
I20260925 16:38:22.380189 2120 catalog_manager.cc:5495] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194). New cstate: current_term: 1 leader_uuid: "6115b4e9f93b46499d54ac4f56f5861b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6115b4e9f93b46499d54ac4f56f5861b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:22.439689 2558 raft_consensus.cc:1261] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:22.439755 2278 raft_consensus.cc:1261] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Refusing update from remote peer 6115b4e9f93b46499d54ac4f56f5861b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:22.440100 2720 consensus_queue.cc:1035] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "328f1a52705e4f5181f59ef3115e5739" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:22.440255 2733 consensus_queue.cc:1035] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [LEADER]: Connected to new peer: Peer: permanent_uuid: "675292cae7394b0fb005a935f0c0e932" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:38:22.453018 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:38:24.008857 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:24.009038 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:24.011483 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c223d1cf1f45e1ba928058fa480597: Sending DeleteTablet for 3 replicas of tablet adaee7214cf141e9a3ca9e2b1379904f
I20260925 16:38:24.012059 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet adaee7214cf141e9a3ca9e2b1379904f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:24 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:47696
I20260925 16:38:24.012159 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet adaee7214cf141e9a3ca9e2b1379904f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:24 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 16:38:24.012485 2761 tablet_replica.cc:330] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: stopping tablet replica
I20260925 16:38:24.012614 2761 raft_consensus.cc:2229] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:24.012790 2761 raft_consensus.cc:2258] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:24.013005 2762 tablet_replica.cc:330] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: stopping tablet replica
I20260925 16:38:24.013118 2762 raft_consensus.cc:2229] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:24.013240 2762 raft_consensus.cc:2258] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:24.013845 2762 ts_tablet_manager.cc:1905] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:24.016492 2762 ts_tablet_manager.cc:1918] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:24.016571 2762 log.cc:1196] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/adaee7214cf141e9a3ca9e2b1379904f
I20260925 16:38:24.016826 2762 ts_tablet_manager.cc:1939] T adaee7214cf141e9a3ca9e2b1379904f P 328f1a52705e4f5181f59ef3115e5739: Deleting consensus metadata
I20260925 16:38:24.017208 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet adaee7214cf141e9a3ca9e2b1379904f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:24 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42642
I20260925 16:38:24.017619 2763 tablet_replica.cc:330] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: stopping tablet replica
I20260925 16:38:24.017666 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2193
I20260925 16:38:24.018033 2761 ts_tablet_manager.cc:1905] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:24.018507 2763 raft_consensus.cc:2229] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:24.018710 2763 raft_consensus.cc:2258] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:24.019400 2763 ts_tablet_manager.cc:1905] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:24.020433 2761 ts_tablet_manager.cc:1918] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:24.020529 2761 log.cc:1196] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/adaee7214cf141e9a3ca9e2b1379904f
I20260925 16:38:24.020798 2761 ts_tablet_manager.cc:1939] T adaee7214cf141e9a3ca9e2b1379904f P 6115b4e9f93b46499d54ac4f56f5861b: Deleting consensus metadata
I20260925 16:38:24.021560 2107 catalog_manager.cc:4841] TS 6115b4e9f93b46499d54ac4f56f5861b (127.0.97.194:46009): tablet adaee7214cf141e9a3ca9e2b1379904f (table test-table [id=d23bb4ed8ca2433f9142795472e0bd8f]) successfully deleted
I20260925 16:38:24.022188 2763 ts_tablet_manager.cc:1918] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:38:24.022249 2763 log.cc:1196] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/adaee7214cf141e9a3ca9e2b1379904f
I20260925 16:38:24.022504 2763 ts_tablet_manager.cc:1939] T adaee7214cf141e9a3ca9e2b1379904f P 675292cae7394b0fb005a935f0c0e932: Deleting consensus metadata
I20260925 16:38:24.023152 2108 catalog_manager.cc:4841] TS 675292cae7394b0fb005a935f0c0e932 (127.0.97.195:39435): tablet adaee7214cf141e9a3ca9e2b1379904f (table test-table [id=d23bb4ed8ca2433f9142795472e0bd8f]) successfully deleted
W20260925 16:38:24.027415 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:43105 (error 108)
I20260925 16:38:24.027700 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2333
I20260925 16:38:24.029424 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:24.028777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:43105 (local address 127.0.0.1:50050)
0925 16:38:24.029079 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:24.029168 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
W20260925 16:38:24.029709 2108 catalog_manager.cc:4584] TS 328f1a52705e4f5181f59ef3115e5739 (127.0.97.193:43105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adaee7214cf141e9a3ca9e2b1379904f: Network error: Client connection negotiation failed: client connection to 127.0.97.193:43105: connect: Connection refused (error 111)
I20260925 16:38:24.035108 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2473
I20260925 16:38:24.040679 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2089
2026-09-25T16:38:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7007 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:24 dist-test-slave-kxl8 krb5kdc[2768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:24 dist-test-slave-kxl8 krb5kdc[2768](info): set up 2 sockets
Sep 25 16:38:24 dist-test-slave-kxl8 krb5kdc[2768](info): commencing operation
krb5kdc: starting...
W20260925 16:38:25.097231 391 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:25.217149 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36955
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:25.326793 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:25.327069 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:25.327162 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:25.331197 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:25.331254 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:25.331272 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:25.331288 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:25.331331 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:25.336514 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:25.337594 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:25.338642 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:25.347258 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:25.347271 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:25.347348 2781 server_base.cc:1034] running on GCE node
W20260925 16:38:25.347280 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:25.348085 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:25.349249 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:25.350476 2781 hybrid_clock.cc:648] HybridClock initialized: now 1790354305350464 us; error 27 us; skew 500 ppm
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:25.357844 2781 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:25.358733 2781 webserver.cc:460] Webserver started at http://127.0.97.254:42181/ using document root <none> and password file <none>
I20260925 16:38:25.359329 2781 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:25.359419 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:25.359642 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:25.361706 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "62ac15c09ad6452691551612c1918fa5"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "9d301ac98c9b4fd01f4959e607e1b98a"
server_key_iv: "193ab01b70a0a5f2f59342b2f48bbd9e"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.362286 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "62ac15c09ad6452691551612c1918fa5"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "9d301ac98c9b4fd01f4959e607e1b98a"
server_key_iv: "193ab01b70a0a5f2f59342b2f48bbd9e"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.365823 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:25.368141 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.369486 2781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:25.369616 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "62ac15c09ad6452691551612c1918fa5"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "9d301ac98c9b4fd01f4959e607e1b98a"
server_key_iv: "193ab01b70a0a5f2f59342b2f48bbd9e"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.369720 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:25.390206 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:25.393532 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:25.393738 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:25.400542 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36955
I20260925 16:38:25.400665 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36955 every 8 connection(s)
I20260925 16:38:25.401603 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:25.403008 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2781
I20260925 16:38:25.403134 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:25.403421 391 external_mini_cluster.cc:1428] Setting key b71a30e3a6b165fa356373cc2dcb93a0
I20260925 16:38:25.404903 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:25.410295 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5: Bootstrap starting.
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:25.412460 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:25.413359 2849 log.cc:824] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:25.415295 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5: No bootstrap required, opened a new log
I20260925 16:38:25.416999 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:25.404776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.97.254:36955)
0925 16:38:25.405182 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:25.405197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:25.405240 (+ 43us) server_negotiation.cc:407] Connection header received
0925 16:38:25.405909 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:25.405925 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:25.406243 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:25.406577 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:25.407347 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.408317 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.409179 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.409710 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.409875 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:25.412338 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:25.412343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:25.412355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:25.412381 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:25.414281 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.414704 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.414710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.414721 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.414827 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.415106 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.415111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.415113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.415584 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:25.415768 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:25.416059 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 16:38:25.416337 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260925 16:38:25.419142 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } }
I20260925 16:38:25.419401 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:25.419482 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ac15c09ad6452691551612c1918fa5, State: Initialized, Role: FOLLOWER
I20260925 16:38:25.419914 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } }
I20260925 16:38:25.420053 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:25.420114 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:25.420220 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:25.421068 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } }
I20260925 16:38:25.421427 2849 leader_election.cc:302] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ac15c09ad6452691551612c1918fa5; no voters:
I20260925 16:38:25.421730 2849 leader_election.cc:288] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:25.421878 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:25.422125 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 62ac15c09ad6452691551612c1918fa5, State: Running, Role: LEADER
I20260925 16:38:25.422410 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } }
I20260925 16:38:25.422920 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:25.424028 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ac15c09ad6452691551612c1918fa5. Latest consensus state: current_term: 1 leader_uuid: "62ac15c09ad6452691551612c1918fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } } }
I20260925 16:38:25.424007 2855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ac15c09ad6452691551612c1918fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ac15c09ad6452691551612c1918fa5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36955 } } }
I20260925 16:38:25.424173 2855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:25.424173 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:25.424558 2863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:25.427666 2863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:25.433018 2863 catalog_manager.cc:1347] Generated new cluster ID: 3c79c6bfd0d84ab68338bb8f4930a3ca
I20260925 16:38:25.433084 2863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:25.441571 2863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:25.442528 2863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:25.446760 2863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ac15c09ad6452691551612c1918fa5: Generated new TSK 0
I20260925 16:38:25.447386 2863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:25.480185 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:25.588877 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:25.589167 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:25.589244 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:25.593214 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:25.593319 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:25.593438 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:25.598744 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:25.599799 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:25.600835 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:25.610621 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:25.610648 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:25.610822 2875 server_base.cc:1034] running on GCE node
W20260925 16:38:25.610648 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:25.611294 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:25.611994 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:25.613193 2875 hybrid_clock.cc:648] HybridClock initialized: now 1790354305613177 us; error 36 us; skew 500 ppm
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:25.616264 2875 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:25.617110 2875 webserver.cc:460] Webserver started at http://127.0.97.193:45003/ using document root <none> and password file <none>
I20260925 16:38:25.617771 2875 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:25.617871 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:25.618106 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:25.620102 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "c17a98c6ae914089aa571f2826cdbec8"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "ed02b90f39ddf0ba64e414de35277e0a"
server_key_iv: "996c83d833aedf1d1f8961781b7f6e37"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.620675 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "c17a98c6ae914089aa571f2826cdbec8"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "ed02b90f39ddf0ba64e414de35277e0a"
server_key_iv: "996c83d833aedf1d1f8961781b7f6e37"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.624915 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 16:38:25.627537 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.628779 2875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:25.628922 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "c17a98c6ae914089aa571f2826cdbec8"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "ed02b90f39ddf0ba64e414de35277e0a"
server_key_iv: "996c83d833aedf1d1f8961781b7f6e37"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.629036 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:25.650429 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:25.654037 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:25.654253 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:25.654855 2875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:25.655702 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:25.655772 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.655845 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:25.655896 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.664444 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:45201
I20260925 16:38:25.664516 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:45201 every 8 connection(s)
I20260925 16:38:25.665468 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:25.666538 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 2875
I20260925 16:38:25.666618 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:25.666958 391 external_mini_cluster.cc:1428] Setting key c728932513f7da904ece3ef41f0d5420
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:25.680413 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:25.667338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:50869 (local address 127.0.97.254:36955)
0925 16:38:25.667459 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:25.667463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:25.668438 (+ 975us) server_negotiation.cc:407] Connection header received
0925 16:38:25.669976 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:25.669979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:25.670033 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:25.670116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:25.671307 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.671844 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.673054 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.673355 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.673467 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:25.676373 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:25.676376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:25.676379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:25.676405 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:25.677975 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.678457 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.678460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.678461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.678579 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.679082 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.679086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.679088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.679367 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:25.679493 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:25.679971 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 16:38:25.680140 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:38:25.681303 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:25.667823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36955 (local address 127.0.97.193:50869)
0925 16:38:25.668296 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:25.668348 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:25.669770 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:25.670307 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:25.670315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:25.670663 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:25.671152 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:25.671174 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.671970 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.671974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:25.672933 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:25.672942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.673506 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.673510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:25.673674 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:25.674384 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:25.674399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:25.676193 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:25.678099 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:25.678104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:25.678116 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:25.678366 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:25.678778 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:25.678785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:25.678790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:25.678972 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:25.679483 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:25.679488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:25.679715 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 16:38:25.679879 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 16:38:25.680133 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260925 16:38:25.682663 3004 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36955
I20260925 16:38:25.682945 3004 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:25.683475 3004 heartbeater.cc:511] Master 127.0.97.254:36955 requested a full tablet report, sending...
I20260925 16:38:25.685120 2812 ts_manager.cc:194] Registered new tserver with Master: c17a98c6ae914089aa571f2826cdbec8 (127.0.97.193:45201)
I20260925 16:38:25.686348 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:50869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:25.697340 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:25.806882 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:25.807171 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:25.807247 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:25.811338 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:25.811421 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:25.811512 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:25.816684 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:25.817775 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:25.818764 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:25.828290 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:25.828333 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:25.828290 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:25.828600 3009 server_base.cc:1034] running on GCE node
I20260925 16:38:25.829034 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:25.829743 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:25.830919 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790354305830899 us; error 33 us; skew 500 ppm
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:25.833854 3009 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:25.834712 3009 webserver.cc:460] Webserver started at http://127.0.97.194:39673/ using document root <none> and password file <none>
I20260925 16:38:25.835258 3009 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:25.835314 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:25.835477 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:25.837483 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "28c65b5b6e1d45fea3a36ec337271f97"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "d6bc70d08de2ea5f4a6d7a5357859cf9"
server_key_iv: "14f859bbc075181c19d55a287973075f"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.837983 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "28c65b5b6e1d45fea3a36ec337271f97"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "d6bc70d08de2ea5f4a6d7a5357859cf9"
server_key_iv: "14f859bbc075181c19d55a287973075f"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.841565 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:25.843853 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.845034 3009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:25.845135 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "28c65b5b6e1d45fea3a36ec337271f97"
format_stamp: "Formatted at 2026-09-25 16:38:25 on dist-test-slave-kxl8"
server_key: "d6bc70d08de2ea5f4a6d7a5357859cf9"
server_key_iv: "14f859bbc075181c19d55a287973075f"
server_key_version: "encryptionkey@0"
I20260925 16:38:25.845322 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:25.868827 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:25.872124 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:25.872332 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:25.872948 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:25.873843 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:25.873890 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.873929 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:25.873986 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:25.882655 3009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38537
I20260925 16:38:25.882781 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38537 every 8 connection(s)
I20260925 16:38:25.883610 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
Sep 25 16:38:25 dist-test-slave-kxl8 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:25.893523 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3009
I20260925 16:38:25.893644 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:25.893925 391 external_mini_cluster.cc:1428] Setting key fc965afaa7c8c075604750797dafb6d3
I20260925 16:38:25.898428 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:25.885447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:52521 (local address 127.0.97.254:36955)
0925 16:38:25.885611 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:25.885616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:25.886494 (+ 878us) server_negotiation.cc:407] Connection header received
0925 16:38:25.888030 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:25.888033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:25.888125 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:25.888219 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:25.889523 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.890122 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.891217 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.891450 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.891538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:25.894294 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:25.894297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:25.894300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:25.894326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:25.895983 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.896628 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.896631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.896632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.896713 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:25.897174 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:25.897179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:25.897183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:25.897457 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:25.897565 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:25.898048 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 16:38:25.898192 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:38:25.899070 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:25.885850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36955 (local address 127.0.97.194:52521)
0925 16:38:25.886319 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:25.886374 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:25.887804 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:25.888438 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:25.888444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:25.888808 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:25.889365 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:25.889382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.890233 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.890235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:25.891099 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:25.891107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:25.891580 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:25.891583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:25.891713 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:25.892322 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:25.892338 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:25.894109 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:25.896109 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:25.896118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:25.896136 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:25.896511 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:25.896805 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:25.896811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:25.896816 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:25.897006 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:25.897559 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:25.897564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:25.897792 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 16:38:25.897941 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:38:25.898188 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
I20260925 16:38:25.900264 3138 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36955
I20260925 16:38:25.900489 3138 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:25.900966 3138 heartbeater.cc:511] Master 127.0.97.254:36955 requested a full tablet report, sending...
I20260925 16:38:25.902011 2812 ts_manager.cc:194] Registered new tserver with Master: 28c65b5b6e1d45fea3a36ec337271f97 (127.0.97.194:38537)
I20260925 16:38:25.902549 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:52521
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:25.921923 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:26.033675 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:26.034013 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:26.034098 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:26.038357 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:26.038440 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:26.038527 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:26.043867 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:26.045074 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:26.046137 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:26.055850 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:26.055856 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:26.055899 3143 server_base.cc:1034] running on GCE node
W20260925 16:38:26.055847 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:26.056418 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:26.057068 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:26.058269 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790354306058260 us; error 59 us; skew 500 ppm
Sep 25 16:38:26 dist-test-slave-kxl8 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:26.061204 3143 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:26.062131 3143 webserver.cc:460] Webserver started at http://127.0.97.195:36319/ using document root <none> and password file <none>
I20260925 16:38:26.062682 3143 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:26.062769 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:26.062996 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:26.064944 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "d836968f63544de3b08ee4429ada28a3"
format_stamp: "Formatted at 2026-09-25 16:38:26 on dist-test-slave-kxl8"
server_key: "fd6f725cfb9eb98b0764bc3717333a8d"
server_key_iv: "4c9c87ead33cc2084b137c65a01e4146"
server_key_version: "encryptionkey@0"
I20260925 16:38:26.065526 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "d836968f63544de3b08ee4429ada28a3"
format_stamp: "Formatted at 2026-09-25 16:38:26 on dist-test-slave-kxl8"
server_key: "fd6f725cfb9eb98b0764bc3717333a8d"
server_key_iv: "4c9c87ead33cc2084b137c65a01e4146"
server_key_version: "encryptionkey@0"
I20260925 16:38:26.069072 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:38:26.071452 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:26.072688 3143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:26.072827 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "d836968f63544de3b08ee4429ada28a3"
format_stamp: "Formatted at 2026-09-25 16:38:26 on dist-test-slave-kxl8"
server_key: "fd6f725cfb9eb98b0764bc3717333a8d"
server_key_iv: "4c9c87ead33cc2084b137c65a01e4146"
server_key_version: "encryptionkey@0"
I20260925 16:38:26.072930 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:26.113245 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:26.116793 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:26.116951 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:26.117503 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:26.118304 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:26.118347 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:26.118386 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:26.118402 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:26.127048 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:36699
I20260925 16:38:26.127080 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:36699 every 8 connection(s)
I20260925 16:38:26.127954 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:26.128197 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3143
I20260925 16:38:26.128331 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:26.128609 391 external_mini_cluster.cc:1428] Setting key d7455876d1b493a12d4e961d3d1910a7
Sep 25 16:38:26 dist-test-slave-kxl8 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:26.142036 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:26.129728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:42687 (local address 127.0.97.254:36955)
0925 16:38:26.129873 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:26.129878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:26.130900 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 16:38:26.132322 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:26.132325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:26.132379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:26.132450 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:26.133736 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:26.134249 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:26.135395 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:26.135608 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:26.135690 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:26.138078 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:26.138082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:26.138085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:26.138110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:26.139728 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:26.140293 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:26.140297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:26.140299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:26.140396 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:26.140746 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:26.140752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:26.140754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:26.140983 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:26.141100 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:26.141724 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 16:38:26.141855 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:38:26.142905 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:26.130257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36955 (local address 127.0.97.195:42687)
0925 16:38:26.130761 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:26.130815 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:26.132026 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:26.132602 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:26.132608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:26.133014 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:26.133606 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:26.133621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:26.134368 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:26.134371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:26.135280 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:26.135288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:26.135731 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:26.135734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:26.135869 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:26.136405 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:26.136419 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:26.137924 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:26.139890 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:26.139896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:26.139910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:26.140181 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:26.140502 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:26.140505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:26.140507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:26.140645 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:26.141095 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:26.141102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:26.141449 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 16:38:26.141670 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 16:38:26.141993 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260925 16:38:26.143982 3272 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36955
I20260925 16:38:26.144205 3272 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:26.144665 3272 heartbeater.cc:511] Master 127.0.97.254:36955 requested a full tablet report, sending...
I20260925 16:38:26.145665 2812 ts_manager.cc:194] Registered new tserver with Master: d836968f63544de3b08ee4429ada28a3 (127.0.97.195:36699)
I20260925 16:38:26.146230 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:42687
I20260925 16:38:26.154191 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:38:26.166728 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:26.165165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.0.97.254:36955)
0925 16:38:26.165338 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:26.165343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:26.165521 (+ 178us) server_negotiation.cc:407] Connection header received
0925 16:38:26.165738 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:26.165741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:26.165783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:26.165865 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:26.166578 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39882: BlockingRecv error: recv got EOF from 127.0.0.1:39882 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:26.167903 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2875
I20260925 16:38:26.173650 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3009
I20260925 16:38:26.178524 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3143
I20260925 16:38:26.183741 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 2781
2026-09-25T16:38:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2141 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:26 dist-test-slave-kxl8 krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:26 dist-test-slave-kxl8 krb5kdc[3284](info): set up 2 sockets
Sep 25 16:38:26 dist-test-slave-kxl8 krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260925 16:38:27.225324 391 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:27.345588 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38469
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:33985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:27.453415 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:27.453678 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:27.453719 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:27.457620 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:27.457679 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:27.457700 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:27.457717 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:27.457732 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:27.462844 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:27.463868 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:27.464856 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:27.474009 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:27.473982 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:27.473984 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:27.474041 3297 server_base.cc:1034] running on GCE node
I20260925 16:38:27.474852 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:27.476027 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:27.477205 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790354307477185 us; error 30 us; skew 500 ppm
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:27.480250 3297 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:27.481120 3297 webserver.cc:460] Webserver started at http://127.0.97.254:41013/ using document root <none> and password file <none>
I20260925 16:38:27.481716 3297 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:27.481778 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:27.481988 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:27.484025 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "7c0f9b44264f46ca8853ae3ba7841cc3"
format_stamp: "Formatted at 2026-09-25 16:38:27 on dist-test-slave-kxl8"
server_key: "847fd7d2cb8d287ee2e7781fd5f32f9e"
server_key_iv: "0e4be93395a698f1161b82c2cebe9f59"
server_key_version: "encryptionkey@0"
I20260925 16:38:27.484637 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "7c0f9b44264f46ca8853ae3ba7841cc3"
format_stamp: "Formatted at 2026-09-25 16:38:27 on dist-test-slave-kxl8"
server_key: "847fd7d2cb8d287ee2e7781fd5f32f9e"
server_key_iv: "0e4be93395a698f1161b82c2cebe9f59"
server_key_version: "encryptionkey@0"
I20260925 16:38:27.488410 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:27.491007 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:27.492386 3297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:27.492507 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "7c0f9b44264f46ca8853ae3ba7841cc3"
format_stamp: "Formatted at 2026-09-25 16:38:27 on dist-test-slave-kxl8"
server_key: "847fd7d2cb8d287ee2e7781fd5f32f9e"
server_key_iv: "0e4be93395a698f1161b82c2cebe9f59"
server_key_version: "encryptionkey@0"
I20260925 16:38:27.492611 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:27.509644 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:27.512982 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:27.513382 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:27.520746 3297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:38469
I20260925 16:38:27.521028 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:38469 every 8 connection(s)
I20260925 16:38:27.521896 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:27.524665 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:27.529987 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3: Bootstrap starting.
I20260925 16:38:27.531327 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3297
I20260925 16:38:27.531471 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:27.531762 391 external_mini_cluster.cc:1428] Setting key ae55fdf8e1a70254c8cd5235ffd905b4
I20260925 16:38:27.532600 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:27.533499 3365 log.cc:824] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:27.535591 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3: No bootstrap required, opened a new log
I20260925 16:38:27.539781 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } }
I20260925 16:38:27.539989 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:27.540023 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0f9b44264f46ca8853ae3ba7841cc3, State: Initialized, Role: FOLLOWER
I20260925 16:38:27.540481 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } }
I20260925 16:38:27.540621 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:27.540692 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:27.540795 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:27.541888 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } }
I20260925 16:38:27.542250 3365 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0f9b44264f46ca8853ae3ba7841cc3; no voters:
I20260925 16:38:27.542608 3365 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:27.542717 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:27.542974 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0f9b44264f46ca8853ae3ba7841cc3, State: Running, Role: LEADER
I20260925 16:38:27.543329 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } }
I20260925 16:38:27.543972 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:27.544414 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0f9b44264f46ca8853ae3ba7841cc3. Latest consensus state: current_term: 1 leader_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } } }
I20260925 16:38:27.544452 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f9b44264f46ca8853ae3ba7841cc3" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38469 } } }
I20260925 16:38:27.544552 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:27.544587 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:27.544937 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:27.547834 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:27.532831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.97.254:38469)
0925 16:38:27.533377 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:27.533396 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:27.533438 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:38:27.534202 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:27.534224 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:27.534615 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:27.534980 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:27.535636 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:27.537127 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:27.538056 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:27.538826 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:27.539023 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:27.541885 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:27.541891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:27.541909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:27.541940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:27.543989 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:27.544536 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:27.544542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:27.544547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:27.544710 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:27.545106 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:27.545110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:27.545112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:27.545648 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:27.545849 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:27.546578 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 16:38:27.546903 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":161,"threads_started":1}
I20260925 16:38:27.548290 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:27.550918 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:38:27.554203 3374 catalog_manager.cc:1347] Generated new cluster ID: e0a7b5ed7ce947159039ad2158bdef0b
I20260925 16:38:27.554265 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:27.571061 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:27.572268 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:27.573115 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:27.567714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.0.97.254:38469)
0925 16:38:27.567835 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:27.567839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:27.568031 (+ 192us) server_negotiation.cc:407] Connection header received
0925 16:38:27.568150 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:27.568154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:27.568206 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:27.568306 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:27.568873 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:27.569691 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:27.570668 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:27.571019 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:27.571114 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:27.572051 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:27.572055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:27.572058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:27.572612 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:27.572654 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:27.572673 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:27.572762 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:27.572900 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 16:38:27.572976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260925 16:38:27.580448 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0f9b44264f46ca8853ae3ba7841cc3: Generated new TSK 0
I20260925 16:38:27.581107 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:38:27.587698 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3297
2026-09-25T16:38:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1401 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3398](info): set up 2 sockets
Sep 25 16:38:27 dist-test-slave-kxl8 krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260925 16:38:28.649170 391 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:28 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:28.770627 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36045
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:28.883462 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:28.883775 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:28.883852 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:28.888564 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:28.888667 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:28.888705 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:28.888772 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:28.888828 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:28.895207 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:28.896567 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:28.897820 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:28.906880 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:28.906880 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:28.906975 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:28.907145 3411 server_base.cc:1034] running on GCE node
I20260925 16:38:28.907776 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:28.908967 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:28.910130 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790354308910095 us; error 48 us; skew 500 ppm
Sep 25 16:38:28 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:28.913000 3411 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:28.913987 3411 webserver.cc:460] Webserver started at http://127.0.97.254:35995/ using document root <none> and password file <none>
I20260925 16:38:28.914546 3411 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:28.914608 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:28.914848 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:28.916945 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "382994cdab9b43d6bbadaa0f396e7074"
format_stamp: "Formatted at 2026-09-25 16:38:28 on dist-test-slave-kxl8"
server_key: "00fbe8174b1e47d81c3ec12ee8a4b5d8"
server_key_iv: "4fdd71ac9846e4ddcd0b3b2bec6b72da"
server_key_version: "encryptionkey@0"
I20260925 16:38:28.917522 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "382994cdab9b43d6bbadaa0f396e7074"
format_stamp: "Formatted at 2026-09-25 16:38:28 on dist-test-slave-kxl8"
server_key: "00fbe8174b1e47d81c3ec12ee8a4b5d8"
server_key_iv: "4fdd71ac9846e4ddcd0b3b2bec6b72da"
server_key_version: "encryptionkey@0"
I20260925 16:38:28.921207 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:38:28.923836 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:28.925130 3411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 16:38:28.925259 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "382994cdab9b43d6bbadaa0f396e7074"
format_stamp: "Formatted at 2026-09-25 16:38:28 on dist-test-slave-kxl8"
server_key: "00fbe8174b1e47d81c3ec12ee8a4b5d8"
server_key_iv: "4fdd71ac9846e4ddcd0b3b2bec6b72da"
server_key_version: "encryptionkey@0"
I20260925 16:38:28.925407 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:28.954187 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:28.957412 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:28.957640 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:28.964566 3411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36045
I20260925 16:38:28.964941 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36045 every 8 connection(s)
I20260925 16:38:28.965642 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:28.966611 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3411
I20260925 16:38:28.966778 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:28.967051 391 external_mini_cluster.cc:1428] Setting key 2ad1c23d61346df23614eb04c28e9ff2
I20260925 16:38:28.969165 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 16:38:28 dist-test-slave-kxl8 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:28.976505 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074: Bootstrap starting.
I20260925 16:38:28.979406 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:28.980396 3479 log.cc:824] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:28.982167 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:28.968390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.97.254:36045)
0925 16:38:28.968841 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:28.968862 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:28.968906 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:38:28.969780 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:28.969804 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:28.970204 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:28.970563 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:28.971231 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:28.972759 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:28.973675 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:28.974364 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:28.974523 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:28.977076 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:28.977082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:28.977098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:28.977129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:28.979115 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:28.979564 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:28.979570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:28.979576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:28.979732 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:28.980027 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:28.980030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:28.980032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:28.980560 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:28.980745 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:28.981049 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 16:38:28.981394 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260925 16:38:28.982626 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074: No bootstrap required, opened a new log
I20260925 16:38:28.986353 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } }
I20260925 16:38:28.986601 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:28.986685 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382994cdab9b43d6bbadaa0f396e7074, State: Initialized, Role: FOLLOWER
I20260925 16:38:28.987177 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } }
I20260925 16:38:28.987329 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:28.987417 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:28.987514 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:29.026926 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } }
I20260925 16:38:29.027441 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382994cdab9b43d6bbadaa0f396e7074; no voters:
I20260925 16:38:29.027830 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:29.027977 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:29.028247 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [term 1 LEADER]: Becoming Leader. State: Replica: 382994cdab9b43d6bbadaa0f396e7074, State: Running, Role: LEADER
I20260925 16:38:29.028707 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } }
I20260925 16:38:29.029493 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:29.030601 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382994cdab9b43d6bbadaa0f396e7074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } } }
I20260925 16:38:29.030730 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:29.031070 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382994cdab9b43d6bbadaa0f396e7074. Latest consensus state: current_term: 1 leader_uuid: "382994cdab9b43d6bbadaa0f396e7074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382994cdab9b43d6bbadaa0f396e7074" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36045 } } }
I20260925 16:38:29.031217 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:29.031548 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:29.034351 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:29.039431 3493 catalog_manager.cc:1347] Generated new cluster ID: f464c71f8b39481e88160a151272c450
I20260925 16:38:29.039500 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:29.058156 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:29.059232 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:29.064167 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382994cdab9b43d6bbadaa0f396e7074: Generated new TSK 0
I20260925 16:38:29.064859 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:29.149147 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:29.260931 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:29.261247 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:29.261346 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:29.265507 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:29.265596 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:29.265693 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:29.270918 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:29.271961 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:29.272982 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:38:29.282626 3505 server_base.cc:1034] running on GCE node
W20260925 16:38:29.282575 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.282586 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.282554 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:29.283098 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:29.283777 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:29.284967 3505 hybrid_clock.cc:648] HybridClock initialized: now 1790354309284955 us; error 26 us; skew 500 ppm
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:29.287747 3505 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:29.288611 3505 webserver.cc:460] Webserver started at http://127.0.97.193:38827/ using document root <none> and password file <none>
I20260925 16:38:29.289182 3505 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:29.289273 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:29.289521 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:29.291530 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "c18560c3c01a493796c76b2f04b185f4"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "bb5a4464c1a122d5eccf7432ccfd2684"
server_key_iv: "8129acb9a887a0eec455dab1263f5168"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.292136 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "c18560c3c01a493796c76b2f04b185f4"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "bb5a4464c1a122d5eccf7432ccfd2684"
server_key_iv: "8129acb9a887a0eec455dab1263f5168"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.296080 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 16:38:29.298599 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.299845 3505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:29.299980 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "c18560c3c01a493796c76b2f04b185f4"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "bb5a4464c1a122d5eccf7432ccfd2684"
server_key_iv: "8129acb9a887a0eec455dab1263f5168"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.300091 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:29.326750 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:29.329808 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:29.330021 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:29.330691 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:29.331568 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:29.331636 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.331704 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:29.331755 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.340543 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:37571
I20260925 16:38:29.340569 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:37571 every 8 connection(s)
I20260925 16:38:29.341478 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:29.345973 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3505
I20260925 16:38:29.346091 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:29.346377 391 external_mini_cluster.cc:1428] Setting key 91706e4eeb8b08ffc6e55e18e6d70cae
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:29.357265 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.343340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:43247 (local address 127.0.97.254:36045)
0925 16:38:29.343481 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:29.343485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:29.344325 (+ 840us) server_negotiation.cc:407] Connection header received
0925 16:38:29.345679 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:29.345682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:29.345731 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:29.345795 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:29.347346 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.348050 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.349899 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.350254 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.350379 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.353415 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:29.353418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:29.353421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:29.353454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:29.354955 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.355571 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.355573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.355575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.355649 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.356052 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.356054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.356055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.356237 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:29.356312 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:29.357009 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 16:38:29.357126 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:29.358322 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.343722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36045 (local address 127.0.97.193:43247)
0925 16:38:29.344149 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:29.344209 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:29.345485 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:29.346005 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:29.346015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:29.346553 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:29.347205 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.347221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.348205 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.348209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.349776 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.349786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.350428 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.350432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.350601 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.351222 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:29.351239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:29.353099 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:29.355075 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.355083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.355101 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.355466 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.355741 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.355744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.355746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.355946 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.356349 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:29.356357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:29.356672 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 16:38:29.356965 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 16:38:29.357366 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:29.359602 3634 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36045
I20260925 16:38:29.359864 3634 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:29.360464 3634 heartbeater.cc:511] Master 127.0.97.254:36045 requested a full tablet report, sending...
I20260925 16:38:29.362344 3443 ts_manager.cc:194] Registered new tserver with Master: c18560c3c01a493796c76b2f04b185f4 (127.0.97.193:37571)
I20260925 16:38:29.363566 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:43247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:29.374449 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:29.482808 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:29.483065 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:29.483107 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:29.487097 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:29.487156 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:29.487223 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:29.492480 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:29.493538 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:29.494478 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:38:29.504216 3639 server_base.cc:1034] running on GCE node
W20260925 16:38:29.504194 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.504179 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.504179 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:29.504813 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:29.505561 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:29.506775 3639 hybrid_clock.cc:648] HybridClock initialized: now 1790354309506733 us; error 30 us; skew 500 ppm
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:29.509511 3639 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:29.510425 3639 webserver.cc:460] Webserver started at http://127.0.97.194:35479/ using document root <none> and password file <none>
I20260925 16:38:29.510988 3639 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:29.511049 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:29.511296 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:29.513405 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "3dbdebc6ca834d9fa453c373cbc69b97"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "e0849faf901c75194df3a26d7603f133"
server_key_iv: "fbbf92bbe2caa55426f498ebb3b55193"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.513978 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "3dbdebc6ca834d9fa453c373cbc69b97"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "e0849faf901c75194df3a26d7603f133"
server_key_iv: "fbbf92bbe2caa55426f498ebb3b55193"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.517627 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:38:29.519968 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.521095 3639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:29.521245 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "3dbdebc6ca834d9fa453c373cbc69b97"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "e0849faf901c75194df3a26d7603f133"
server_key_iv: "fbbf92bbe2caa55426f498ebb3b55193"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.521400 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:29.535897 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:29.538708 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:29.538904 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:29.539499 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:29.540385 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:29.540432 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.540499 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:29.540539 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.549424 3639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:34739
I20260925 16:38:29.549456 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:34739 every 8 connection(s)
I20260925 16:38:29.550325 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:29.560532 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3639
I20260925 16:38:29.560621 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:29.560904 391 external_mini_cluster.cc:1428] Setting key caaeb585ba365f3367d988475c29db19
I20260925 16:38:29.563961 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.552116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:50259 (local address 127.0.97.254:36045)
0925 16:38:29.552258 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:29.552262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:29.553076 (+ 814us) server_negotiation.cc:407] Connection header received
0925 16:38:29.554467 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:29.554470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:29.554526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:29.554632 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:29.555762 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.556281 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.557459 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.557681 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.557799 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.560185 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:29.560188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:29.560190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:29.560213 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:29.561867 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.562426 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.562430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.562432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.562515 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.562874 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.562877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.562879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.563058 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:29.563171 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:29.563664 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 16:38:29.563805 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:38:29.564869 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.552498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36045 (local address 127.0.97.194:50259)
0925 16:38:29.552927 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:29.552968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:29.554252 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:29.554805 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:29.554812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:29.555153 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:29.555617 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.555630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.556418 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.556421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.557347 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.557356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.557830 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.557833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.558003 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.558614 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:29.558630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:29.560035 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:29.562005 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.562010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.562025 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.562305 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.562615 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.562618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.562620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.562767 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.563168 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:29.563175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:29.563427 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 16:38:29.563619 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 16:38:29.563959 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20260925 16:38:29.566188 3768 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36045
I20260925 16:38:29.566447 3768 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:29.566939 3768 heartbeater.cc:511] Master 127.0.97.254:36045 requested a full tablet report, sending...
I20260925 16:38:29.568037 3443 ts_manager.cc:194] Registered new tserver with Master: 3dbdebc6ca834d9fa453c373cbc69b97 (127.0.97.194:34739)
I20260925 16:38:29.568868 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:50259
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:29.588264 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:29.698632 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:29.698894 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:29.698938 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:29.703405 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:29.703469 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:29.703575 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:29.709084 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:29.710129 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:29.711171 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:29.720649 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.720649 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:29.720623 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:29.720686 3773 server_base.cc:1034] running on GCE node
I20260925 16:38:29.721330 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:29.721998 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:29.723201 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790354309723186 us; error 27 us; skew 500 ppm
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:29.726146 3773 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:29.726934 3773 webserver.cc:460] Webserver started at http://127.0.97.195:34127/ using document root <none> and password file <none>
I20260925 16:38:29.727461 3773 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:29.727519 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:29.727676 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:29.729858 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "e5b1de02b1bc45989e0c4a5aa6038753"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "983bcf28093d3eb40341d35ad288c221"
server_key_iv: "33a0c8791142998e73536a80d5fe448a"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.730453 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "e5b1de02b1bc45989e0c4a5aa6038753"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "983bcf28093d3eb40341d35ad288c221"
server_key_iv: "33a0c8791142998e73536a80d5fe448a"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.734179 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:38:29.736573 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.737848 3773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:29.737967 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "e5b1de02b1bc45989e0c4a5aa6038753"
format_stamp: "Formatted at 2026-09-25 16:38:29 on dist-test-slave-kxl8"
server_key: "983bcf28093d3eb40341d35ad288c221"
server_key_iv: "33a0c8791142998e73536a80d5fe448a"
server_key_version: "encryptionkey@0"
I20260925 16:38:29.738070 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:29.760340 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:29.763783 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:29.763996 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:29.764623 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:29.765527 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:29.765574 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.765640 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:29.765673 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:29.774595 3773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37935
I20260925 16:38:29.774621 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37935 every 8 connection(s)
I20260925 16:38:29.775527 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:29.784458 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3773
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:29.784588 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:29.784907 391 external_mini_cluster.cc:1428] Setting key b211e5022317149e296bf970f8a2e80b
I20260925 16:38:29.788765 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.777366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:42293 (local address 127.0.97.254:36045)
0925 16:38:29.777535 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:29.777539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:29.778428 (+ 889us) server_negotiation.cc:407] Connection header received
0925 16:38:29.779733 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:29.779736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:29.779783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:29.779863 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:29.780965 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.781520 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.782579 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.782785 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.782860 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.785347 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:29.785352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:29.785355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:29.785387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:29.787096 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.787537 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.787539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.787540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.787610 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.787897 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.787899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.787900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.788064 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:29.788137 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:29.788544 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 16:38:29.788646 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:29.789582 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.777848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36045 (local address 127.0.97.195:42293)
0925 16:38:29.778278 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:29.778320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:29.779530 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:29.780021 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:29.780027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:29.780383 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:29.780842 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.780855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.781624 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.781626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.782477 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:29.782486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.782904 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.782907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:29.783035 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.783577 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:29.783592 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:29.785111 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:29.787194 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.787199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.787209 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.787455 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.787685 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:29.787687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:29.787688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:29.787823 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:29.788145 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:29.788150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:29.788356 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 16:38:29.788499 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 16:38:29.788740 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260925 16:38:29.790548 3902 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36045
I20260925 16:38:29.790788 3902 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:29.791249 3902 heartbeater.cc:511] Master 127.0.97.254:36045 requested a full tablet report, sending...
I20260925 16:38:29.792236 3443 ts_manager.cc:194] Registered new tserver with Master: e5b1de02b1bc45989e0c4a5aa6038753 (127.0.97.195:37935)
I20260925 16:38:29.792801 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:42293
I20260925 16:38:29.798856 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:29.823137 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:29.815260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.97.254:36045)
0925 16:38:29.815458 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:29.815461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:29.815524 (+ 63us) server_negotiation.cc:407] Connection header received
0925 16:38:29.815630 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:29.815632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:29.815675 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:29.815763 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:29.816422 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.816936 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.817764 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:29.817973 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:29.818054 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:29.820315 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:29.820318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:29.820320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:29.820342 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:29.821649 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.822019 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.822021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.822022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.822093 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:29.822307 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:29.822309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:29.822310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:29.822513 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:29.822615 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:29.822799 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:38:29.822940 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 16:38:29.823568 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43736
I20260925 16:38:29.824955 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3505
I20260925 16:38:29.831539 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3639
I20260925 16:38:29.837080 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3773
I20260925 16:38:29.842248 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3411
2026-09-25T16:38:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2256 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3914](info): set up 2 sockets
Sep 25 16:38:29 dist-test-slave-kxl8 krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260925 16:38:30.933255 391 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:31.056509 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:43393
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:43393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:31.166571 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:31.166836 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:31.166875 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:31.170814 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:31.170867 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:31.170886 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:31.170903 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:31.170951 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:31.176129 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:43393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:31.177140 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:31.178179 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:31.186561 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.186563 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.186607 3927 server_base.cc:1034] running on GCE node
W20260925 16:38:31.186561 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.187404 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:31.188452 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:31.189608 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790354311189585 us; error 35 us; skew 500 ppm
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:31.196184 3927 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:31.197103 3927 webserver.cc:460] Webserver started at http://127.0.97.254:40577/ using document root <none> and password file <none>
I20260925 16:38:31.197788 3927 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:31.197858 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:31.198098 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:31.200179 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "17aec6e58d174d15880f0737fff62b1b"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "620a673d7b0a3de8259f00f490255e8c"
server_key_iv: "5ca36eb79e220b3b8efbf3b65d70b9d8"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.200742 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "17aec6e58d174d15880f0737fff62b1b"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "620a673d7b0a3de8259f00f490255e8c"
server_key_iv: "5ca36eb79e220b3b8efbf3b65d70b9d8"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.204464 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:31.206897 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.208104 3927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:31.208259 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "17aec6e58d174d15880f0737fff62b1b"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "620a673d7b0a3de8259f00f490255e8c"
server_key_iv: "5ca36eb79e220b3b8efbf3b65d70b9d8"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.208369 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:31.220830 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:31.224097 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:31.224301 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:31.231442 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:43393
I20260925 16:38:31.231456 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:43393 every 8 connection(s)
I20260925 16:38:31.232394 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:31.235034 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:31.240144 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b: Bootstrap starting.
I20260925 16:38:31.242514 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:31.242545 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 3927
I20260925 16:38:31.242686 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:31.242976 391 external_mini_cluster.cc:1428] Setting key 48204d17512017c20fb52adeba0f74a6
I20260925 16:38:31.243310 3995 log.cc:824] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:31.246057 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b: No bootstrap required, opened a new log
I20260925 16:38:31.250224 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } }
I20260925 16:38:31.250514 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:31.250584 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17aec6e58d174d15880f0737fff62b1b, State: Initialized, Role: FOLLOWER
I20260925 16:38:31.251012 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } }
I20260925 16:38:31.251164 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:31.251250 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:31.251375 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:31.252259 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } }
I20260925 16:38:31.252594 3995 leader_election.cc:302] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17aec6e58d174d15880f0737fff62b1b; no voters:
I20260925 16:38:31.252940 3995 leader_election.cc:288] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:31.253054 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:31.253382 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [term 1 LEADER]: Becoming Leader. State: Replica: 17aec6e58d174d15880f0737fff62b1b, State: Running, Role: LEADER
I20260925 16:38:31.253756 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } }
I20260925 16:38:31.254402 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:31.254992 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17aec6e58d174d15880f0737fff62b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } } }
I20260925 16:38:31.254992 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17aec6e58d174d15880f0737fff62b1b. Latest consensus state: current_term: 1 leader_uuid: "17aec6e58d174d15880f0737fff62b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aec6e58d174d15880f0737fff62b1b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43393 } } }
I20260925 16:38:31.255218 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:31.255450 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:31.255753 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:31.258767 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.244001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.97.254:43393)
0925 16:38:31.244363 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.244378 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.244411 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:38:31.245195 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.245216 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.245623 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.245964 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.246700 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.248233 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.249217 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.249944 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.250097 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.252744 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:31.252751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:31.252766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:31.252796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:31.254953 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.255547 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.255552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.255558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.255701 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.256025 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.256028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.256030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.256608 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:31.257403 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:31.257568 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:38:31.257849 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260925 16:38:31.259402 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:31.264505 4007 catalog_manager.cc:1347] Generated new cluster ID: 66842594ce2b485aaef86ad070f64bf5
I20260925 16:38:31.264570 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:31.272720 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:31.274147 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:31.280054 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17aec6e58d174d15880f0737fff62b1b: Generated new TSK 0
I20260925 16:38:31.280699 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:31.309116 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:31.420635 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:31.420899 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:31.420943 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:31.424975 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:31.425034 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:31.425091 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:31.430256 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:31.431324 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:31.432349 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:31.441758 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.441763 4026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.441789 4027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.442054 4021 server_base.cc:1034] running on GCE node
I20260925 16:38:31.442444 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:31.443112 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:31.444286 4021 hybrid_clock.cc:648] HybridClock initialized: now 1790354311444268 us; error 31 us; skew 500 ppm
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:31.446977 4021 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:31.447791 4021 webserver.cc:460] Webserver started at http://127.0.97.193:39383/ using document root <none> and password file <none>
I20260925 16:38:31.448344 4021 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:31.448441 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:31.448681 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:31.450752 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "0f9e49f2742b42bbb2bae555a8dfb862"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "4198a26c86e1022437ebe71d93677b32"
server_key_iv: "d1051dc7977ffb566a0535f151e09b31"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.451318 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "0f9e49f2742b42bbb2bae555a8dfb862"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "4198a26c86e1022437ebe71d93677b32"
server_key_iv: "d1051dc7977ffb566a0535f151e09b31"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.455111 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:38:31.457677 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.458886 4021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:31.459015 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "0f9e49f2742b42bbb2bae555a8dfb862"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "4198a26c86e1022437ebe71d93677b32"
server_key_iv: "d1051dc7977ffb566a0535f151e09b31"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.459127 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:31.484615 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:31.487852 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:31.488092 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:31.488737 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:31.489686 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:31.489759 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.489825 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:31.489877 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.498687 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:33577
I20260925 16:38:31.498708 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:33577 every 8 connection(s)
I20260925 16:38:31.499612 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:31.505967 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4021
I20260925 16:38:31.506110 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:31.506394 391 external_mini_cluster.cc:1428] Setting key 6bb28846accb280e1dc1cd37b94d5118
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:31.514364 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.501556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:50677 (local address 127.0.97.254:43393)
0925 16:38:31.501717 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.501722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.502763 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 16:38:31.504075 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.504078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.504123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.504198 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.505435 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.506002 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.507592 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.507920 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.508029 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.510651 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:31.510655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:31.510656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:31.510678 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:31.512195 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.512739 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.512744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.512750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.512854 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.513213 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.513219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.513224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.513467 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:31.513570 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:31.514081 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 16:38:31.514202 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:38:31.515204 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.502029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:43393 (local address 127.0.97.193:50677)
0925 16:38:31.502607 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:31.502651 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:31.503883 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:31.504373 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:31.504379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:31.504722 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:31.505192 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.505204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.506160 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.506163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.507448 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.507458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.508064 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.508067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.508204 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.508861 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:31.508876 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:31.510483 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:31.512319 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.512325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.512340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.512624 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.512965 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.512968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.512970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.513118 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.513580 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:31.513588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:31.513843 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 16:38:31.514032 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:38:31.514369 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:31.516496 4150 heartbeater.cc:347] Connected to a master server at 127.0.97.254:43393
I20260925 16:38:31.516747 4150 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:31.517253 4150 heartbeater.cc:511] Master 127.0.97.254:43393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:31.518883 3959 ts_manager.cc:194] Registered new tserver with Master: 0f9e49f2742b42bbb2bae555a8dfb862 (127.0.97.193:33577)
I20260925 16:38:31.519981 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:50677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:31.534420 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:31.643853 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:31.644107 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:31.644151 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:31.648258 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:31.648319 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:31.648411 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:31.653733 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:31.654861 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:31.655923 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:31.665653 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.665637 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.665637 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.665936 4155 server_base.cc:1034] running on GCE node
I20260925 16:38:31.666373 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:31.666963 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:31.668167 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790354311668150 us; error 30 us; skew 500 ppm
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:31.671052 4155 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:31.671911 4155 webserver.cc:460] Webserver started at http://127.0.97.194:33079/ using document root <none> and password file <none>
I20260925 16:38:31.672436 4155 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:31.672495 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:31.672657 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:31.674751 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "1609febc3ba648e783915bae08da26e5"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "e89f4f33527555efacfc77de0af080d3"
server_key_iv: "9647615a60897aff86c54dbdf938cef4"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.675261 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "1609febc3ba648e783915bae08da26e5"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "e89f4f33527555efacfc77de0af080d3"
server_key_iv: "9647615a60897aff86c54dbdf938cef4"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.678943 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:31.681340 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.682560 4155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:31.682657 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "1609febc3ba648e783915bae08da26e5"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "e89f4f33527555efacfc77de0af080d3"
server_key_iv: "9647615a60897aff86c54dbdf938cef4"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.682762 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:31.701213 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:31.704363 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:31.704512 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:31.705040 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:31.705993 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:31.706046 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.706087 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:31.706107 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.715641 4155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:46465
I20260925 16:38:31.715660 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:46465 every 8 connection(s)
I20260925 16:38:31.716562 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:31.720546 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4155
I20260925 16:38:31.720715 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:31.721016 391 external_mini_cluster.cc:1428] Setting key c2b56519785f7fc586d65df420daaaf9
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:31.731334 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.718498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37143 (local address 127.0.97.254:43393)
0925 16:38:31.718643 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.718647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.719544 (+ 897us) server_negotiation.cc:407] Connection header received
0925 16:38:31.721057 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.721060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.721115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.721230 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.722762 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.723282 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.724426 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.724650 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.724764 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.727548 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:31.727551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:31.727554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:31.727575 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:31.729030 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.729653 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.729656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.729658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.729741 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.730135 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.730138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.730140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.730333 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:31.730441 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:31.731025 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 16:38:31.731148 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:38:31.732218 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.718901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:43393 (local address 127.0.97.194:37143)
0925 16:38:31.719378 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:31.719423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:31.720842 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:31.721437 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:31.721446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:31.721948 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:31.722615 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.722631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.723405 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.723408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.724300 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.724309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.724810 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.724814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.724990 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.725669 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:31.725685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:31.727363 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:31.729169 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.729175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.729192 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.729545 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.729872 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.729875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.729877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.730028 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.730439 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:31.730446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:31.730750 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 16:38:31.730969 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 16:38:31.731305 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:38:31.733495 4284 heartbeater.cc:347] Connected to a master server at 127.0.97.254:43393
I20260925 16:38:31.733747 4284 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:31.734241 4284 heartbeater.cc:511] Master 127.0.97.254:43393 requested a full tablet report, sending...
I20260925 16:38:31.735304 3959 ts_manager.cc:194] Registered new tserver with Master: 1609febc3ba648e783915bae08da26e5 (127.0.97.194:46465)
I20260925 16:38:31.735873 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:37143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:31.748821 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:31.856827 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:31.857067 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:31.857146 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:31.861200 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:31.861255 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:31.861326 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:31.866542 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:31.867537 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:31.868525 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:31.877964 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.878085 4289 server_base.cc:1034] running on GCE node
W20260925 16:38:31.877955 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:31.877967 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:31.878536 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:31.879195 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:31.880395 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790354311880383 us; error 26 us; skew 500 ppm
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:31.883225 4289 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:31.884152 4289 webserver.cc:460] Webserver started at http://127.0.97.195:45513/ using document root <none> and password file <none>
I20260925 16:38:31.884714 4289 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:31.884800 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:31.885025 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:31.887161 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "c44512db76dd4a1b80d7c15ecdf65b46"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "cb967b1687fca721f0f22ae8147fedc1"
server_key_iv: "228b6c279392768d2938bea2a85a382f"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.887728 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "c44512db76dd4a1b80d7c15ecdf65b46"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "cb967b1687fca721f0f22ae8147fedc1"
server_key_iv: "228b6c279392768d2938bea2a85a382f"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.891292 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:31.893749 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.895152 4289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:31.895303 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "c44512db76dd4a1b80d7c15ecdf65b46"
format_stamp: "Formatted at 2026-09-25 16:38:31 on dist-test-slave-kxl8"
server_key: "cb967b1687fca721f0f22ae8147fedc1"
server_key_iv: "228b6c279392768d2938bea2a85a382f"
server_key_version: "encryptionkey@0"
I20260925 16:38:31.895430 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:31.931739 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:31.935531 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:31.935796 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:31.936461 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:31.937381 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:31.937458 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.937537 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:31.937599 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:31.946848 4289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:42161
I20260925 16:38:31.946872 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:42161 every 8 connection(s)
I20260925 16:38:31.947803 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:31.955466 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4289
I20260925 16:38:31.955597 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:31.955884 391 external_mini_cluster.cc:1428] Setting key e1bc513cadd68d0bdad800c23e55c7eb
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:31.961630 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.949833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:53061 (local address 127.0.97.254:43393)
0925 16:38:31.950001 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.950005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.950709 (+ 704us) server_negotiation.cc:407] Connection header received
0925 16:38:31.952143 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.952146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.952200 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.952321 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.953520 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.954061 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.955360 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.955608 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.955723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.958555 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:31.958558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:31.958560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:31.958583 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:31.959803 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.960253 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.960255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.960257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.960327 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.960607 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.960609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.960610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.960791 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:31.960867 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:31.961355 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 16:38:31.961488 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:38:31.962525 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.950090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:43393 (local address 127.0.97.195:53061)
0925 16:38:31.950539 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:31.950582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:31.951923 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:31.952492 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:31.952498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:31.952861 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:31.953368 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.953382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.954192 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.954229 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.955118 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:31.955127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.955785 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.955789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:31.955990 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.956666 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:31.956687 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:31.958410 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:31.959905 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.959910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.959921 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.960167 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.960403 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:31.960405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:31.960407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:31.960532 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:31.960875 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:31.960880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:31.961094 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 16:38:31.961257 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 16:38:31.961648 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
I20260925 16:38:31.963644 4418 heartbeater.cc:347] Connected to a master server at 127.0.97.254:43393
I20260925 16:38:31.963923 4418 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:31.964411 4418 heartbeater.cc:511] Master 127.0.97.254:43393 requested a full tablet report, sending...
I20260925 16:38:31.965485 3959 ts_manager.cc:194] Registered new tserver with Master: c44512db76dd4a1b80d7c15ecdf65b46 (127.0.97.195:42161)
I20260925 16:38:31.966099 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:53061
I20260925 16:38:31.969691 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:38:31 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:31.995425 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.986370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.97.193:33577)
0925 16:38:31.986712 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.986726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.986758 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:38:31.986809 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.986813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.987064 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.987184 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.987849 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.988600 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.989521 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.989910 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:31.989992 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:31.992329 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:31.992337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:31.992350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:31.992372 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:31.993886 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.994273 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.994278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.994279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.994354 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:31.994581 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:31.994584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:31.994585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:31.994831 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:31.994997 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:31.995157 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 16:38:31.995276 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":135,"threads_started":1}
Sep 25 16:38:32 dist-test-slave-kxl8 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:32.006263 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:31.998317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.0.97.254:43393)
0925 16:38:31.998454 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:31.998458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:31.998471 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:31.998555 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:31.998557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:31.998595 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:31.998660 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:31.999209 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:31.999787 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:32.000514 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:32.000767 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:32.000845 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:32.003079 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:32.003082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:32.003084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:32.003106 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:32.004591 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:32.005005 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:32.005009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:32.005011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:32.005088 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:32.005389 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:32.005393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:32.005395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:32.005606 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:32.005751 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:32.005979 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 16:38:32.006095 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 16:38:32.006667 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50808
I20260925 16:38:32.007321 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4021
I20260925 16:38:32.012964 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4155
I20260925 16:38:32.018699 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4289
I20260925 16:38:32.023907 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 3927
2026-09-25T16:38:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2181 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:32 dist-test-slave-kxl8 krb5kdc[4430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:32 dist-test-slave-kxl8 krb5kdc[4430](info): set up 2 sockets
Sep 25 16:38:32 dist-test-slave-kxl8 krb5kdc[4430](info): commencing operation
krb5kdc: starting...
W20260925 16:38:33.081831 391 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:33.204030 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45305
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:33.314333 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:33.314651 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:33.314723 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:33.318809 4443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:33.318894 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:33.318919 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:33.318939 4443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:33.318977 4443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:33.324257 4443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:33.325325 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:33.326359 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:33.334604 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:33.334607 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:33.334852 4443 server_base.cc:1034] running on GCE node
W20260925 16:38:33.334566 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:33.335631 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:33.336781 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:33.337994 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790354313337979 us; error 24 us; skew 500 ppm
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:33.340981 4443 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:33.341768 4443 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:33.341867 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:33.342093 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:33.344080 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "a222e7cac6354974816a6f445f59ba46"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "286034cb583c6d857a1777593ac33079"
server_key_iv: "875fa2335b592dea6f2b9ca180a745b0"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.344630 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "a222e7cac6354974816a6f445f59ba46"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "286034cb583c6d857a1777593ac33079"
server_key_iv: "875fa2335b592dea6f2b9ca180a745b0"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.348188 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:33.350551 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.351724 4443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:33.351877 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "a222e7cac6354974816a6f445f59ba46"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "286034cb583c6d857a1777593ac33079"
server_key_iv: "875fa2335b592dea6f2b9ca180a745b0"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.351986 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:33.378914 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:33.382403 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:33.382620 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:33.389639 4443 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:45305
I20260925 16:38:33.390027 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:45305 every 8 connection(s)
I20260925 16:38:33.390337 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:33.393111 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:33.398437 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: Bootstrap starting.
I20260925 16:38:33.400273 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4443
I20260925 16:38:33.400424 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:33.400673 391 external_mini_cluster.cc:1428] Setting key 024a1ee1721647af503d5d7310e91a53
I20260925 16:38:33.400933 4510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:33.401958 4510 log.cc:824] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:33.404013 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: No bootstrap required, opened a new log
I20260925 16:38:33.408499 4510 raft_consensus.cc:348] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } }
I20260925 16:38:33.408834 4510 raft_consensus.cc:374] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:33.408893 4510 raft_consensus.cc:729] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a222e7cac6354974816a6f445f59ba46, State: Initialized, Role: FOLLOWER
I20260925 16:38:33.409435 4510 consensus_queue.cc:260] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } }
I20260925 16:38:33.409569 4510 raft_consensus.cc:388] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:33.409654 4510 raft_consensus.cc:482] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:33.409750 4510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:33.410642 4510 raft_consensus.cc:504] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } }
I20260925 16:38:33.410984 4510 leader_election.cc:302] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a222e7cac6354974816a6f445f59ba46; no voters:
I20260925 16:38:33.411334 4510 leader_election.cc:288] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:33.411445 4515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:33.411613 4515 raft_consensus.cc:686] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [term 1 LEADER]: Becoming Leader. State: Replica: a222e7cac6354974816a6f445f59ba46, State: Running, Role: LEADER
I20260925 16:38:33.411911 4515 consensus_queue.cc:237] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } }
I20260925 16:38:33.412628 4510 sys_catalog.cc:564] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:33.413076 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a222e7cac6354974816a6f445f59ba46. Latest consensus state: current_term: 1 leader_uuid: "a222e7cac6354974816a6f445f59ba46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } } }
I20260925 16:38:33.413197 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:33.413520 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a222e7cac6354974816a6f445f59ba46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a222e7cac6354974816a6f445f59ba46" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45305 } } }
I20260925 16:38:33.413620 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:33.413653 4519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:33.416690 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:33.401783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.0.97.254:45305)
0925 16:38:33.402179 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:33.402206 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:33.402241 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:38:33.403005 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:33.403025 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:33.403348 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:33.403618 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:33.404555 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.405573 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.406743 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.407266 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.407396 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:33.410363 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:33.410373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:33.410387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:33.410413 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:33.412490 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.413049 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.413055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.413062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.413402 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.413831 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.413834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.413835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.414354 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:33.415272 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:33.415437 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:38:33.415745 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260925 16:38:33.417022 4519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:33.422947 4519 catalog_manager.cc:1347] Generated new cluster ID: d07f060b89794ef1830ae41580b21d08
I20260925 16:38:33.423018 4519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:33.433995 4519 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:33.435209 4519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:33.448324 4519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: Generated new TSK 0
I20260925 16:38:33.449182 4519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:33.479861 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:33.590595 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:33.590863 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:33.590909 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:33.594957 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:33.595014 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:33.595073 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:33.600294 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:33.601465 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:33.602505 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:33.611532 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:33.611573 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:33.611577 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:33.611845 4536 server_base.cc:1034] running on GCE node
I20260925 16:38:33.612246 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:33.612846 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:33.614025 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790354313613954 us; error 92 us; skew 500 ppm
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:33.616848 4536 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:33.617573 4536 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:33.617635 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:33.617801 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:33.619704 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "fe1eef44ab0c431ebb7447a886976916"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "e123374c4a70ef653bfbe942354206ee"
server_key_iv: "dd67ebadd39e76bd8bc221e7ab13c486"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.620211 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "fe1eef44ab0c431ebb7447a886976916"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "e123374c4a70ef653bfbe942354206ee"
server_key_iv: "dd67ebadd39e76bd8bc221e7ab13c486"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.623708 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:33.625981 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.627163 4536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:33.627256 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "fe1eef44ab0c431ebb7447a886976916"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "e123374c4a70ef653bfbe942354206ee"
server_key_iv: "dd67ebadd39e76bd8bc221e7ab13c486"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.627334 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:33.648227 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:33.651602 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:33.651751 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:33.652268 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:33.653136 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:33.653184 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.653223 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:33.653244 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.662017 4536 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:35595
I20260925 16:38:33.662042 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:35595 every 8 connection(s)
I20260925 16:38:33.662588 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:33.666286 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4536
I20260925 16:38:33.666424 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:33.666716 391 external_mini_cluster.cc:1428] Setting key cb091d66605ac54f11d1c3681f682cc4
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:33.677155 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:33.664308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:38197 (local address 127.0.97.254:45305)
0925 16:38:33.664442 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:33.664446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:33.665308 (+ 862us) server_negotiation.cc:407] Connection header received
0925 16:38:33.666697 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:33.666700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:33.666755 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:33.666835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:33.668331 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.668868 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.670561 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.670895 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.671024 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:33.673474 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:33.673481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:33.673486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:33.673518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:33.675154 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.675646 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.675650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.675653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.675751 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.676087 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.676091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.676093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.676328 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:33.676444 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:33.676935 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 16:38:33.677036 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:38:33.677975 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:33.664711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45305 (local address 127.0.97.193:38197)
0925 16:38:33.665141 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:33.665182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:33.666462 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:33.667000 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:33.667007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:33.667502 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:33.668169 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:33.668189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.669002 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.669006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:33.670390 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:33.670403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.671045 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.671048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:33.671187 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:33.671798 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:33.671814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:33.673252 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:33.675264 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:33.675269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:33.675280 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:33.675529 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:33.675851 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:33.675853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:33.675855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:33.675979 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:33.676446 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:33.676452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:33.676693 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 16:38:33.676853 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 16:38:33.677095 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:33.679572 4664 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45305
I20260925 16:38:33.679837 4664 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:33.680279 4664 heartbeater.cc:511] Master 127.0.97.254:45305 requested a full tablet report, sending...
I20260925 16:38:33.681805 4474 ts_manager.cc:194] Registered new tserver with Master: fe1eef44ab0c431ebb7447a886976916 (127.0.97.193:35595)
I20260925 16:38:33.682904 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:38197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:33.694578 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:33.802718 4669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:33.803018 4669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:33.803099 4669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:33.807044 4669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:33.807132 4669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:33.807224 4669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:33.812325 4669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:33.813345 4669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:33.814350 4669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:38:33.823482 4669 server_base.cc:1034] running on GCE node
W20260925 16:38:33.823446 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:33.823427 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:33.823450 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:33.824067 4669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:33.824690 4669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:33.825841 4669 hybrid_clock.cc:648] HybridClock initialized: now 1790354313825810 us; error 48 us; skew 500 ppm
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:33.828416 4669 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:33.829164 4669 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:33.829253 4669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:33.829535 4669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:33.831626 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "38786bf3748543c4a81d950307f49c1c"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "92c51431f99b9f294b2452daacc53dfd"
server_key_iv: "ecd94703ec35b8d935a5941c9677d717"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.832165 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "38786bf3748543c4a81d950307f49c1c"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "92c51431f99b9f294b2452daacc53dfd"
server_key_iv: "ecd94703ec35b8d935a5941c9677d717"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.835633 4669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 16:38:33.837886 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.839044 4669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:33.839169 4669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "38786bf3748543c4a81d950307f49c1c"
format_stamp: "Formatted at 2026-09-25 16:38:33 on dist-test-slave-kxl8"
server_key: "92c51431f99b9f294b2452daacc53dfd"
server_key_iv: "ecd94703ec35b8d935a5941c9677d717"
server_key_version: "encryptionkey@0"
I20260925 16:38:33.839272 4669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:33.850162 4669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:33.853190 4669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:33.853410 4669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:33.853940 4669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:33.854694 4669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:33.854772 4669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.854866 4669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:33.854916 4669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:33.863706 4669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:36631
I20260925 16:38:33.863727 4796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:36631 every 8 connection(s)
I20260925 16:38:33.864261 4669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:33.870146 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4669
I20260925 16:38:33.870276 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:33.870555 391 external_mini_cluster.cc:1428] Setting key b8ef3e1bd3b1b503610e78f086ef17d7
Sep 25 16:38:33 dist-test-slave-kxl8 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:33.878079 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:33.865930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37961 (local address 127.0.97.254:45305)
0925 16:38:33.866067 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:33.866070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:33.866833 (+ 763us) server_negotiation.cc:407] Connection header received
0925 16:38:33.868119 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:33.868123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:33.868175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:33.868260 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:33.869431 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.870018 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.871399 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.871621 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.871733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:33.874377 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:33.874383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:33.874387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:33.874419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:33.875881 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.876506 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.876509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.876510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.876578 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:33.876972 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:33.876974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:33.876975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:33.877133 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:33.877205 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:33.877876 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 16:38:33.877967 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:33.879240 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:33.866287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45305 (local address 127.0.97.194:37961)
0925 16:38:33.866689 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:33.866729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:33.867920 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:33.868429 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:33.868435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:33.868772 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:33.869226 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:33.869239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.870141 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.870145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:33.871281 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:33.871290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:33.871763 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:33.871767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:33.871988 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:33.872555 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:33.872581 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:33.874208 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:33.876001 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:33.876009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:33.876024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:33.876400 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:33.876671 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:33.876675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:33.876677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:33.876873 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:33.877240 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:33.877248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:33.877598 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 16:38:33.877840 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 16:38:33.878194 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260925 16:38:33.880409 4797 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45305
I20260925 16:38:33.880633 4797 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:33.881043 4797 heartbeater.cc:511] Master 127.0.97.254:45305 requested a full tablet report, sending...
I20260925 16:38:33.882071 4474 ts_manager.cc:194] Registered new tserver with Master: 38786bf3748543c4a81d950307f49c1c (127.0.97.194:36631)
I20260925 16:38:33.882630 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:37961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:33.898003 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:34.006125 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:34.006390 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:34.006438 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:34.010557 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:34.010622 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:34.010681 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:34.016090 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:34.017079 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:34.018168 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:34.027427 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.027449 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.027449 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.027581 4802 server_base.cc:1034] running on GCE node
I20260925 16:38:34.028134 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:34.028762 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:34.029944 4802 hybrid_clock.cc:648] HybridClock initialized: now 1790354314029930 us; error 27 us; skew 500 ppm
Sep 25 16:38:34 dist-test-slave-kxl8 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:34.032841 4802 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:34.033608 4802 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:34.033672 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:34.033910 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:34.035900 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "22fe275314c38701e99d578687e48ed2"
server_key_iv: "721af88bf897086974faff4b874516de"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.036448 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "22fe275314c38701e99d578687e48ed2"
server_key_iv: "721af88bf897086974faff4b874516de"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.040091 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 16:38:34.042452 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.043704 4802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:34.043849 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "22fe275314c38701e99d578687e48ed2"
server_key_iv: "721af88bf897086974faff4b874516de"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.043948 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:34.063234 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:34.066205 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:34.066401 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:34.066968 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:34.067780 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:34.067848 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.067919 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:34.067972 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.076506 4802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37777
I20260925 16:38:34.076524 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37777 every 8 connection(s)
I20260925 16:38:34.077109 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:34.083817 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4802
I20260925 16:38:34.083930 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:34.084185 391 external_mini_cluster.cc:1428] Setting key 08d40d793ee9ad2bc3b77dacadcea4f8
Sep 25 16:38:34 dist-test-slave-kxl8 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:34.090592 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.078883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:46695 (local address 127.0.97.254:45305)
0925 16:38:34.079053 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.079057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.079897 (+ 840us) server_negotiation.cc:407] Connection header received
0925 16:38:34.081161 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.081163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.081209 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.081303 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:34.082475 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.082982 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.084351 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.084557 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.084671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.087239 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:34.087242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:34.087244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:34.087267 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:34.088538 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:34.089056 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:34.089065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:34.089066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:34.089134 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:34.089531 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:34.089536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:34.089538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:34.089716 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:34.089827 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.090309 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.090426 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:38:34.091359 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.079274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45305 (local address 127.0.97.195:46695)
0925 16:38:34.079715 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.079779 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.080968 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.081497 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.081504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.081883 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:34.082352 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.082364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.083082 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.083084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.084225 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.084236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.084711 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.084715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.084936 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.085602 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:34.085617 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:34.087077 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:34.088666 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:34.088671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:34.088685 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:34.088942 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:34.089232 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:34.089235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:34.089237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:34.089436 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:34.089825 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:34.089833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:34.090069 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.090247 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.090561 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260925 16:38:34.092402 4930 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45305
I20260925 16:38:34.092639 4930 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:34.093077 4930 heartbeater.cc:511] Master 127.0.97.254:45305 requested a full tablet report, sending...
I20260925 16:38:34.094092 4474 ts_manager.cc:194] Registered new tserver with Master: 1a2810feb35f4f29a0e33cfa3f4ab93b (127.0.97.195:37777)
I20260925 16:38:34.094647 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:46695
I20260925 16:38:34.097908 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:38:34.106995 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.099655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.97.254:45305)
0925 16:38:34.099824 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.099830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.100010 (+ 180us) server_negotiation.cc:407] Connection header received
0925 16:38:34.100156 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.100159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.100201 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.100298 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:34.100954 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.101558 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.102383 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.102614 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.102723 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.104048 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:34.104051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:34.104054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:34.104078 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:34.105338 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:34.105766 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:34.105770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:34.105772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:34.105857 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:34.106124 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:34.106129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:34.106131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:34.106346 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:34.106489 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.106713 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.106833 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:38:34.109737 4474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:34.112826 4474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:34.127578 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.122100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.97.195:37777)
0925 16:38:34.122515 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.122528 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.122550 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:38:34.122627 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.122632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.122858 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.123043 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:34.123330 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124526 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.126202 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.126948 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.127055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.127197 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.127346 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.127438 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":188,"threads_started":1}
I20260925 16:38:34.127569 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.121684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37777 (local address 127.0.0.1:50552)
0925 16:38:34.121964 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.121980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.122066 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.123040 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.123047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.123073 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:34.123203 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.123209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124930 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.124932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.126088 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.126098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.127072 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.127075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.127159 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.127189 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.127302 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.127418 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260925 16:38:34.128393 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.122073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36631 (local address 127.0.0.1:35626)
0925 16:38:34.122325 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.122339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.122423 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.123483 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.123488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.123501 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:34.123618 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.123625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124675 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.124678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.126088 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.126098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.128012 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.128015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.128103 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.128125 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.128205 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.128274 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":121,"thread_start_us":93,"threads_started":1}
I20260925 16:38:34.129037 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.121260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:35595 (local address 127.0.0.1:34112)
0925 16:38:34.121664 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.121716 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.121860 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.123380 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.123383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.123405 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:34.123530 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.123537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124802 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.124806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.126003 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.126011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.128637 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.128640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.128738 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.128780 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.128856 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.128930 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":9,"thread_start_us":75,"threads_started":1}
I20260925 16:38:34.129460 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.121456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.0.97.193:35595)
0925 16:38:34.122760 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.122770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.122790 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:38:34.123051 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.123056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.123236 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.123412 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:34.123618 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124663 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.126122 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.126948 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.127055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.127197 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.129210 (+ 2013us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.129334 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212,"thread_start_us":106,"threads_started":1}
I20260925 16:38:34.130785 4864 tablet_service.cc:1467] Processing CreateTablet for tablet f07457c5c6194daaadba78c304f5b2a0 (DEFAULT_TABLE table=test-table [id=85c28f98ef6147f3842e9cda00312e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:34.131934 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07457c5c6194daaadba78c304f5b2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:34.131934 4598 tablet_service.cc:1467] Processing CreateTablet for tablet f07457c5c6194daaadba78c304f5b2a0 (DEFAULT_TABLE table=test-table [id=85c28f98ef6147f3842e9cda00312e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:34.132936 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07457c5c6194daaadba78c304f5b2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:34.139213 4947 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Bootstrap starting.
I20260925 16:38:34.128470 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.122626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.0.97.194:36631)
0925 16:38:34.123033 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.123045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.123065 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:38:34.123130 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.123135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.123340 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.123732 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:34.123851 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.124533 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.127255 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.127889 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.127993 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.128100 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.128226 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.128292 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":103,"threads_started":1}
I20260925 16:38:34.140522 4948 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Bootstrap starting.
I20260925 16:38:34.141532 4947 tablet_bootstrap.cc:654] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:34.142570 4948 tablet_bootstrap.cc:654] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:34.142230 4731 tablet_service.cc:1467] Processing CreateTablet for tablet f07457c5c6194daaadba78c304f5b2a0 (DEFAULT_TABLE table=test-table [id=85c28f98ef6147f3842e9cda00312e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:34.143149 4947 log.cc:824] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:34.143352 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07457c5c6194daaadba78c304f5b2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:34.143770 4948 log.cc:824] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:34.145076 4947 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: No bootstrap required, opened a new log
I20260925 16:38:34.145270 4947 ts_tablet_manager.cc:1397] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:38:34.148231 4947 raft_consensus.cc:348] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.148511 4947 raft_consensus.cc:374] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:34.148572 4947 raft_consensus.cc:729] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2810feb35f4f29a0e33cfa3f4ab93b, State: Initialized, Role: FOLLOWER
I20260925 16:38:34.149075 4947 consensus_queue.cc:260] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.149075 4951 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Bootstrap starting.
I20260925 16:38:34.150126 4948 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: No bootstrap required, opened a new log
I20260925 16:38:34.150275 4947 ts_tablet_manager.cc:1428] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260925 16:38:34.150370 4948 ts_tablet_manager.cc:1397] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 16:38:34.150635 4930 heartbeater.cc:503] Master 127.0.97.254:45305 was elected leader, sending a full tablet report...
I20260925 16:38:34.151566 4951 tablet_bootstrap.cc:654] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:34.152565 4951 log.cc:824] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:34.153748 4948 raft_consensus.cc:348] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.153999 4948 raft_consensus.cc:374] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:34.154053 4948 raft_consensus.cc:729] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1eef44ab0c431ebb7447a886976916, State: Initialized, Role: FOLLOWER
I20260925 16:38:34.154556 4951 tablet_bootstrap.cc:492] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: No bootstrap required, opened a new log
I20260925 16:38:34.154492 4948 consensus_queue.cc:260] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.154752 4951 ts_tablet_manager.cc:1397] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 16:38:34.155534 4948 ts_tablet_manager.cc:1428] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:38:34.155936 4664 heartbeater.cc:503] Master 127.0.97.254:45305 was elected leader, sending a full tablet report...
I20260925 16:38:34.158545 4951 raft_consensus.cc:348] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.158830 4951 raft_consensus.cc:374] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:34.158891 4951 raft_consensus.cc:729] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38786bf3748543c4a81d950307f49c1c, State: Initialized, Role: FOLLOWER
I20260925 16:38:34.159258 4951 consensus_queue.cc:260] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.159843 4797 heartbeater.cc:503] Master 127.0.97.254:45305 was elected leader, sending a full tablet report...
I20260925 16:38:34.160089 4951 ts_tablet_manager.cc:1428] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260925 16:38:34.163615 4665 tablet.cc:2369] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:34.197808 4954 raft_consensus.cc:482] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:34.198040 4954 raft_consensus.cc:504] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.199199 4954 leader_election.cc:288] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38786bf3748543c4a81d950307f49c1c (127.0.97.194:36631), 1a2810feb35f4f29a0e33cfa3f4ab93b (127.0.97.195:37777)
I20260925 16:38:34.203531 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.199575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:59017 (local address 127.0.97.194:36631)
0925 16:38:34.199744 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.199749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.200085 (+ 336us) server_negotiation.cc:407] Connection header received
0925 16:38:34.200231 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.200234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.200288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.200369 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:34.200629 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.201128 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.202356 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.202883 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.203002 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.203053 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.203318 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.203394 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:38:34.203624 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.199460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36631 (local address 127.0.97.193:59017)
0925 16:38:34.199985 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.199996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.200117 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.200379 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.200382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.200396 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:34.200517 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.200522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.201241 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.201246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.202216 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.202225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.203010 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.203016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.203156 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.203193 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.203279 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.203382 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":208,"thread_start_us":131,"threads_started":1}
I20260925 16:38:34.203930 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.199567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:46753 (local address 127.0.97.195:37777)
0925 16:38:34.199724 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.199730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.199920 (+ 190us) server_negotiation.cc:407] Connection header received
0925 16:38:34.200086 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.200088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.200131 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.200212 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:34.200489 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.200980 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.202670 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.203228 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.203374 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.203417 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.203727 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.203808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:38:34.204293 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.199457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37777 (local address 127.0.97.193:46753)
0925 16:38:34.199826 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.199838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.199963 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.200236 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.200239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.200254 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:34.200380 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.200385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.201088 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.201095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.202534 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:34.202547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.203477 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.203481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:34.203580 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.203605 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.204060 (+ 455us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.204132 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":70,"thread_start_us":100,"threads_started":1}
I20260925 16:38:34.204334 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07457c5c6194daaadba78c304f5b2a0" candidate_uuid: "fe1eef44ab0c431ebb7447a886976916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38786bf3748543c4a81d950307f49c1c" is_pre_election: true
I20260925 16:38:34.204701 4751 raft_consensus.cc:2454] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1eef44ab0c431ebb7447a886976916 in term 0.
I20260925 16:38:34.205365 4554 leader_election.cc:302] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38786bf3748543c4a81d950307f49c1c, fe1eef44ab0c431ebb7447a886976916; no voters:
I20260925 16:38:34.205896 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07457c5c6194daaadba78c304f5b2a0" candidate_uuid: "fe1eef44ab0c431ebb7447a886976916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" is_pre_election: true
I20260925 16:38:34.206300 4954 raft_consensus.cc:2781] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:34.206321 4884 raft_consensus.cc:2454] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1eef44ab0c431ebb7447a886976916 in term 0.
I20260925 16:38:34.206409 4954 raft_consensus.cc:482] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:34.206470 4954 raft_consensus.cc:3037] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:34.232244 4954 raft_consensus.cc:504] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.232795 4954 leader_election.cc:288] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [CANDIDATE]: Term 1 election: Requested vote from peers 38786bf3748543c4a81d950307f49c1c (127.0.97.194:36631), 1a2810feb35f4f29a0e33cfa3f4ab93b (127.0.97.195:37777)
I20260925 16:38:34.233314 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07457c5c6194daaadba78c304f5b2a0" candidate_uuid: "fe1eef44ab0c431ebb7447a886976916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b"
I20260925 16:38:34.233348 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07457c5c6194daaadba78c304f5b2a0" candidate_uuid: "fe1eef44ab0c431ebb7447a886976916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38786bf3748543c4a81d950307f49c1c"
I20260925 16:38:34.233477 4884 raft_consensus.cc:3037] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:34.233516 4751 raft_consensus.cc:3037] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:34.235008 4751 raft_consensus.cc:2454] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1eef44ab0c431ebb7447a886976916 in term 1.
I20260925 16:38:34.235214 4884 raft_consensus.cc:2454] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1eef44ab0c431ebb7447a886976916 in term 1.
I20260925 16:38:34.235456 4554 leader_election.cc:302] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38786bf3748543c4a81d950307f49c1c, fe1eef44ab0c431ebb7447a886976916; no voters:
I20260925 16:38:34.235625 4954 raft_consensus.cc:2781] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:34.235855 4954 raft_consensus.cc:686] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [term 1 LEADER]: Becoming Leader. State: Replica: fe1eef44ab0c431ebb7447a886976916, State: Running, Role: LEADER
I20260925 16:38:34.236271 4954 consensus_queue.cc:237] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } }
I20260925 16:38:34.239930 4474 catalog_manager.cc:5495] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe1eef44ab0c431ebb7447a886976916 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "fe1eef44ab0c431ebb7447a886976916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1eef44ab0c431ebb7447a886976916" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:34.279093 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.275517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.97.193:35595)
0925 16:38:34.275655 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.275658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.275672 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:34.275828 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.275830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.275887 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.275977 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:34.276229 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.276734 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.277611 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:34.277835 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:34.277919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:34.278206 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:34.278634 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:34.278724 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.278806 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.278857 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:38:34.286159 4751 raft_consensus.cc:1261] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 1 FOLLOWER]: Refusing update from remote peer fe1eef44ab0c431ebb7447a886976916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:34.286185 4884 raft_consensus.cc:1261] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 1 FOLLOWER]: Refusing update from remote peer fe1eef44ab0c431ebb7447a886976916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:34.286979 4954 consensus_queue.cc:1035] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2810feb35f4f29a0e33cfa3f4ab93b" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:34.287218 4958 consensus_queue.cc:1035] T f07457c5c6194daaadba78c304f5b2a0 P fe1eef44ab0c431ebb7447a886976916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38786bf3748543c4a81d950307f49c1c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:34.295022 4961 mvcc.cc:205] Tried to move back new op lower bound from 7333291271307038720 to 7333291271115010048. Current Snapshot: MvccSnapshot[applied={T|T < 7333291271307038720}]
I20260925 16:38:34.298511 4963 mvcc.cc:205] Tried to move back new op lower bound from 7333291271307038720 to 7333291271115010048. Current Snapshot: MvccSnapshot[applied={T|T < 7333291271307038720}]
I20260925 16:38:34.298722 4962 mvcc.cc:205] Tried to move back new op lower bound from 7333291271307038720 to 7333291271115010048. Current Snapshot: MvccSnapshot[applied={T|T < 7333291271307038720 or (T in {7333291271307038720})}]
I20260925 16:38:34.303936 4474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:34.304132 4474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:34.306664 4474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a222e7cac6354974816a6f445f59ba46: Sending DeleteTablet for 3 replicas of tablet f07457c5c6194daaadba78c304f5b2a0
I20260925 16:38:34.307484 4864 tablet_service.cc:1514] Processing DeleteTablet for tablet f07457c5c6194daaadba78c304f5b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:34 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50552
I20260925 16:38:34.307567 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet f07457c5c6194daaadba78c304f5b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:34 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:34112
I20260925 16:38:34.307619 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet f07457c5c6194daaadba78c304f5b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:34 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:35626
I20260925 16:38:34.308069 4972 tablet_replica.cc:330] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: stopping tablet replica
I20260925 16:38:34.308439 4972 raft_consensus.cc:2229] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:34.308477 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4536
I20260925 16:38:34.308830 4972 raft_consensus.cc:2258] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:34.310271 4970 tablet_replica.cc:330] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: stopping tablet replica
I20260925 16:38:34.310588 4970 raft_consensus.cc:2229] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:34.310904 4970 raft_consensus.cc:2258] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:34.312141 4972 ts_tablet_manager.cc:1905] T f07457c5c6194daaadba78c304f5b2a0 P 38786bf3748543c4a81d950307f49c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:34.312249 4970 ts_tablet_manager.cc:1905] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:38:34.315506 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:35595 (error 108)
I20260925 16:38:34.316236 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4669
I20260925 16:38:34.316568 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.316245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:35595 (local address 127.0.0.1:34134)
0925 16:38:34.316400 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.316473 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:38:34.317358 4461 catalog_manager.cc:4584] TS fe1eef44ab0c431ebb7447a886976916 (127.0.97.193:35595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f07457c5c6194daaadba78c304f5b2a0: Network error: Client connection negotiation failed: client connection to 127.0.97.193:35595: connect: Connection refused (error 111)
I20260925 16:38:34.370904 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.370456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:35595 (local address 127.0.0.1:34138)
0925 16:38:34.370711 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.370784 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 16:38:34.371125 4970 ts_tablet_manager.cc:1918] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:34.371248 4970 log.cc:1196] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/f07457c5c6194daaadba78c304f5b2a0
I20260925 16:38:34.371821 4970 ts_tablet_manager.cc:1939] T f07457c5c6194daaadba78c304f5b2a0 P 1a2810feb35f4f29a0e33cfa3f4ab93b: Deleting consensus metadata
I20260925 16:38:34.371835 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.371508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:35595 (local address 127.0.0.1:34148)
0925 16:38:34.371697 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.371754 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 16:38:34.374636 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4802
I20260925 16:38:34.375643 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.374684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36631 (local address 127.0.0.1:35640)
0925 16:38:34.375494 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.375557 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725}
I20260925 16:38:34.384801 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4443
2026-09-25T16:38:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2360 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T16:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:34Z Disabled control of system clock
I20260925 16:38:34.415035 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44409
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:38:34.530514 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:34.530817 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:34.530898 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:34.534930 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:34.535024 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:34.535053 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:34.535094 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:34.535133 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:34.539898 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:34.540982 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:34.542415 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:34.550722 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.550722 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.550894 4977 server_base.cc:1034] running on GCE node
W20260925 16:38:34.550706 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.551532 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:34.552644 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:34.553817 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790354314553787 us; error 43 us; skew 500 ppm
I20260925 16:38:34.555593 4977 webserver.cc:460] Webserver started at http://127.0.97.254:42653/ using document root <none> and password file <none>
I20260925 16:38:34.556180 4977 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:34.556241 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:34.556471 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:34.558620 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "2a303c8cb1104f368e4106c914fbca80"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "d766ad249e374a98d91142edefab1427"
server_key_iv: "8bb9467561d9d43291835ce93cd6317d"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.559187 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "2a303c8cb1104f368e4106c914fbca80"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "d766ad249e374a98d91142edefab1427"
server_key_iv: "8bb9467561d9d43291835ce93cd6317d"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.562811 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:34.565092 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.566336 4977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:34.566476 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "2a303c8cb1104f368e4106c914fbca80"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "d766ad249e374a98d91142edefab1427"
server_key_iv: "8bb9467561d9d43291835ce93cd6317d"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.566586 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:34.571063 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:34.571623 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:34.571806 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:34.578544 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44409
I20260925 16:38:34.578557 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44409 every 8 connection(s)
I20260925 16:38:34.579430 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:34.580775 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 4977
I20260925 16:38:34.580896 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:34.581189 391 external_mini_cluster.cc:1428] Setting key fd4c870eb41d60b2f33b68c7c5813e0d
I20260925 16:38:34.583225 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:34.586861 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.582696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.0.97.254:44409)
0925 16:38:34.583417 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.583431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.583465 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:38:34.584091 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.584105 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.584448 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.584778 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:34.585624 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:34.585628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:34.585639 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:34.585649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:34.585701 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:34.585716 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:34.585854 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.585950 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.586194 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":153,"threads_started":1}
I20260925 16:38:34.589486 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: Bootstrap starting.
I20260925 16:38:34.591961 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:34.592753 5044 log.cc:824] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:34.594792 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: No bootstrap required, opened a new log
I20260925 16:38:34.598338 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } }
I20260925 16:38:34.598558 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:34.598646 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a303c8cb1104f368e4106c914fbca80, State: Initialized, Role: FOLLOWER
I20260925 16:38:34.599093 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } }
I20260925 16:38:34.599292 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:34.599363 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:34.599476 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:34.613639 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } }
I20260925 16:38:34.614190 5044 leader_election.cc:302] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a303c8cb1104f368e4106c914fbca80; no voters:
I20260925 16:38:34.614651 5044 leader_election.cc:288] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:34.614781 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:34.615049 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [term 1 LEADER]: Becoming Leader. State: Replica: 2a303c8cb1104f368e4106c914fbca80, State: Running, Role: LEADER
I20260925 16:38:34.615388 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } }
I20260925 16:38:34.616170 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:34.617008 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a303c8cb1104f368e4106c914fbca80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } } }
I20260925 16:38:34.617132 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:34.617560 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a303c8cb1104f368e4106c914fbca80. Latest consensus state: current_term: 1 leader_uuid: "2a303c8cb1104f368e4106c914fbca80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a303c8cb1104f368e4106c914fbca80" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44409 } } }
I20260925 16:38:34.617709 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:34.617826 5054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:34.621016 5054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:34.626576 5054 catalog_manager.cc:1347] Generated new cluster ID: ff2a11c0cd454ace97fed7e06c580f1c
I20260925 16:38:34.626657 5054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:34.645002 5054 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:34.645370 5054 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:34.653786 5054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: Generated new TSK 0
I20260925 16:38:34.654506 5054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:38:34.666523 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--builtin_ntp_servers=127.0.97.212:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:38:34.777900 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:34.778172 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:34.778287 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:34.782420 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:34.782491 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:34.782578 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:34.787317 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:34.788372 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:34.789752 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:34.799196 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.799190 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.799361 5068 server_base.cc:1034] running on GCE node
W20260925 16:38:34.799578 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.799976 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:34.800576 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:34.801764 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790354314801743 us; error 27 us; skew 500 ppm
I20260925 16:38:34.803572 5068 webserver.cc:460] Webserver started at http://127.0.97.193:34415/ using document root <none> and password file <none>
I20260925 16:38:34.804143 5068 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:34.804239 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:34.804500 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:34.806587 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "d99e8436d46b4e579c9af43734d6fcad"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "e6bba7d4ac253fbee758cdfe03e2cbf0"
server_key_iv: "b734b9a0d751e4a634b8db48663ea62b"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.807145 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "d99e8436d46b4e579c9af43734d6fcad"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "e6bba7d4ac253fbee758cdfe03e2cbf0"
server_key_iv: "b734b9a0d751e4a634b8db48663ea62b"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.810833 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:38:34.813118 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.814296 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:34.814433 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "d99e8436d46b4e579c9af43734d6fcad"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "e6bba7d4ac253fbee758cdfe03e2cbf0"
server_key_iv: "b734b9a0d751e4a634b8db48663ea62b"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.814541 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:34.819352 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:34.819921 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:34.820099 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:34.820617 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:34.821559 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:34.821627 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.821694 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:34.821748 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.830673 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:40977
I20260925 16:38:34.830686 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:40977 every 8 connection(s)
I20260925 16:38:34.831638 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:34.832559 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5068
I20260925 16:38:34.832695 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:34.833040 391 external_mini_cluster.cc:1428] Setting key cc918dfe860f1594cd72e7d429c8e1da
I20260925 16:38:34.835006 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--builtin_ntp_servers=127.0.97.212:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:38:34.838587 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.833683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:57511 (local address 127.0.97.254:44409)
0925 16:38:34.833837 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.833842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.835105 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 16:38:34.836654 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:34.836657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:34.836699 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:34.836771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:34.838074 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:34.838077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:34.838080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:34.838092 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:34.838114 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:34.838124 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:34.838225 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:34.838440 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 16:38:34.838477 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:38:34.839845 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.834173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44409 (local address 127.0.97.193:57511)
0925 16:38:34.834947 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.835003 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:34.836429 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:34.837011 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:34.837018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:34.837106 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:34.837811 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:34.837813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:34.837854 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:34.837919 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:34.838239 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:34.838252 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:34.838260 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:38:34.838473 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 16:38:34.838733 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":144,"threads_started":1}
I20260925 16:38:34.841126 5196 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44409
I20260925 16:38:34.841423 5196 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:34.841926 5196 heartbeater.cc:511] Master 127.0.97.254:44409 requested a full tablet report, sending...
I20260925 16:38:34.843806 5008 ts_manager.cc:194] Registered new tserver with Master: d99e8436d46b4e579c9af43734d6fcad (127.0.97.193:40977)
W20260925 16:38:34.944092 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:34.944344 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:34.944382 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:34.948242 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:34.948288 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:34.948344 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:34.953058 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:34.954082 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:34.955379 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:34.964646 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.964632 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:34.964686 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:34.964776 5199 server_base.cc:1034] running on GCE node
I20260925 16:38:34.965199 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:34.965866 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:34.967039 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790354314967026 us; error 27 us; skew 500 ppm
I20260925 16:38:34.968762 5199 webserver.cc:460] Webserver started at http://127.0.97.194:37071/ using document root <none> and password file <none>
I20260925 16:38:34.969379 5199 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:34.969472 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:34.969755 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:34.972033 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "465d1759af5b4cca8f51bba58e7c5af5"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "f5db6be6fb8b5e9fc6cad90a081e3800"
server_key_iv: "20d5e9558748881f9c6e6a8622914894"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.972597 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "465d1759af5b4cca8f51bba58e7c5af5"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "f5db6be6fb8b5e9fc6cad90a081e3800"
server_key_iv: "20d5e9558748881f9c6e6a8622914894"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.976140 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:34.978477 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.979645 5199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:34.979773 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "465d1759af5b4cca8f51bba58e7c5af5"
format_stamp: "Formatted at 2026-09-25 16:38:34 on dist-test-slave-kxl8"
server_key: "f5db6be6fb8b5e9fc6cad90a081e3800"
server_key_iv: "20d5e9558748881f9c6e6a8622914894"
server_key_version: "encryptionkey@0"
I20260925 16:38:34.979885 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:34.984445 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:34.985044 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:34.985229 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:34.985780 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:34.986677 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:34.986747 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.986809 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:34.986858 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:34.995630 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38475
I20260925 16:38:34.995654 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38475 every 8 connection(s)
I20260925 16:38:34.996526 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:35.000901 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5199
I20260925 16:38:35.001020 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:35.001332 391 external_mini_cluster.cc:1428] Setting key dff141ccd1a174b5ece0f3202234122a
I20260925 16:38:35.002655 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.998384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:38287 (local address 127.0.97.254:44409)
0925 16:38:34.998548 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:34.998553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:34.999586 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 16:38:35.000854 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.000856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.000889 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.000953 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.002174 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.002178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.002181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.002193 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.002213 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.002230 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.002338 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.002484 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.002527 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:38:35.003212 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--builtin_ntp_servers=127.0.97.212:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:38:35.003649 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:34.998785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44409 (local address 127.0.97.194:38287)
0925 16:38:34.999433 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:34.999472 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.000601 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.001132 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.001141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.001259 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.001924 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.001926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.001965 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.002030 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.002328 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.002336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.002342 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.002527 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.002878 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260925 16:38:35.004808 5327 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44409
I20260925 16:38:35.005015 5327 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:35.005491 5327 heartbeater.cc:511] Master 127.0.97.254:44409 requested a full tablet report, sending...
I20260925 16:38:35.006505 5008 ts_manager.cc:194] Registered new tserver with Master: 465d1759af5b4cca8f51bba58e7c5af5 (127.0.97.194:38475)
W20260925 16:38:35.119482 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:35.119820 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:35.119894 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:35.124029 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:35.124081 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:35.124166 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:35.128875 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:35.130033 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:35.131476 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:35.140851 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:35.140846 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:35.140951 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:35.141199 5330 server_base.cc:1034] running on GCE node
I20260925 16:38:35.141690 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:35.142309 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:35.143503 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790354315143493 us; error 26 us; skew 500 ppm
I20260925 16:38:35.145219 5330 webserver.cc:460] Webserver started at http://127.0.97.195:33641/ using document root <none> and password file <none>
I20260925 16:38:35.145797 5330 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:35.145888 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:35.146111 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:35.148057 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "6ff8d09f43024bca944347250784a7e3"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9735942a0a0aba0d10520cac2714069e"
server_key_iv: "20133600b1a5d5396ba7e41d3efb07f2"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.148622 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff8d09f43024bca944347250784a7e3"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9735942a0a0aba0d10520cac2714069e"
server_key_iv: "20133600b1a5d5396ba7e41d3efb07f2"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.152199 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:35.154825 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:35.156001 5330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:35.156131 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "6ff8d09f43024bca944347250784a7e3"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9735942a0a0aba0d10520cac2714069e"
server_key_iv: "20133600b1a5d5396ba7e41d3efb07f2"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.156239 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:35.160745 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:35.161389 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:35.161557 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:35.162092 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:35.162925 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:35.162994 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:35.163059 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:35.163107 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:35.172652 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:36469
I20260925 16:38:35.172785 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:36469 every 8 connection(s)
I20260925 16:38:35.173640 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:35.179312 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.175373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:38525 (local address 127.0.97.254:44409)
0925 16:38:35.175507 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.175511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.176509 (+ 998us) server_negotiation.cc:407] Connection header received
0925 16:38:35.177810 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.177812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.177848 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.177921 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.178896 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.178899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.178901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.178912 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.178930 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.178937 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.179010 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.179139 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.179174 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:38:35.179545 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5330
I20260925 16:38:35.179663 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:35.179956 391 external_mini_cluster.cc:1428] Setting key bd1fbe00202090273a7826860d3e2cb4
I20260925 16:38:35.180341 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.175745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44409 (local address 127.0.97.195:38525)
0925 16:38:35.176368 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.176404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.177621 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.178069 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.178075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.178158 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.178663 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.178665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.178701 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.178764 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.179009 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.179015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.179018 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.179156 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.179419 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260925 16:38:35.181484 5458 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44409
I20260925 16:38:35.181697 5458 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:35.182132 5458 heartbeater.cc:511] Master 127.0.97.254:44409 requested a full tablet report, sending...
I20260925 16:38:35.183036 5008 ts_manager.cc:194] Registered new tserver with Master: 6ff8d09f43024bca944347250784a7e3 (127.0.97.195:36469)
I20260925 16:38:35.193699 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:38:35.197091 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.195530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.97.254:44409)
0925 16:38:35.195677 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.195682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.195876 (+ 194us) server_negotiation.cc:407] Connection header received
0925 16:38:35.195953 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.195959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.196001 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.196082 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.196652 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.196656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.196658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.196668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.196685 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.196693 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.196759 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.196801 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.196831 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:35.199609 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:35.202575 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:35.213402 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.210586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40977 (local address 127.0.0.1:56368)
0925 16:38:35.211296 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.211310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.211374 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.212049 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.212052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.212065 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.212589 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.212591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.212603 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.212631 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.213057 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.213066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.213073 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.213165 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.213234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":880,"threads_started":1}
I20260925 16:38:35.213532 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.210441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38475 (local address 127.0.0.1:41518)
0925 16:38:35.210922 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.210966 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.211062 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.211702 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.211708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.211812 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.212375 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.212377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.212396 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.212466 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.213189 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.213193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.213197 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.213326 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.213370 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":80,"threads_started":1}
I20260925 16:38:35.213912 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.210591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.0.97.194:38475)
0925 16:38:35.211205 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.211219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.211244 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:38:35.211310 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.211317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.211538 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.212103 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.212984 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.212992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.213003 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.213016 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.213053 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.213077 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.213210 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.213354 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.213428 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":149,"threads_started":1}
I20260925 16:38:35.214071 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.211056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.97.193:40977)
0925 16:38:35.211605 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.211618 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.211643 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:38:35.211696 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.211701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.211891 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.212244 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.212851 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.212857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.212869 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.212879 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.212917 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.212945 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.213778 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.213836 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.213898 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":136,"threads_started":1}
I20260925 16:38:35.214634 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.211466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.97.195:36469)
0925 16:38:35.212592 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.212606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.212628 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:38:35.212678 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.212684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.212845 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.212979 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.214110 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.214116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.214128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.214138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.214185 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.214206 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.214307 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.214400 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.214459 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":130,"threads_started":1}
I20260925 16:38:35.215123 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.211050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:36469 (local address 127.0.0.1:35546)
0925 16:38:35.211660 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.211673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.211750 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.212964 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.212967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.212980 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.213647 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.213650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.213663 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.213687 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.214289 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.214292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.214295 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.214966 (+ 671us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.215003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":100,"threads_started":1}
I20260925 16:38:35.216763 5130 tablet_service.cc:1467] Processing CreateTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 (DEFAULT_TABLE table=test-table [id=f1322b5e8d6d4e9db592343a0d2eb2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:35.216763 5261 tablet_service.cc:1467] Processing CreateTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 (DEFAULT_TABLE table=test-table [id=f1322b5e8d6d4e9db592343a0d2eb2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:35.217366 5392 tablet_service.cc:1467] Processing CreateTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 (DEFAULT_TABLE table=test-table [id=f1322b5e8d6d4e9db592343a0d2eb2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:35.217875 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b9bac7191e4000b78a6548ceb57ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:35.218039 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b9bac7191e4000b78a6548ceb57ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:35.218322 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b9bac7191e4000b78a6548ceb57ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:35.224073 5475 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Bootstrap starting.
I20260925 16:38:35.225606 5476 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: Bootstrap starting.
I20260925 16:38:35.225607 5477 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Bootstrap starting.
I20260925 16:38:35.226369 5475 tablet_bootstrap.cc:654] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:35.227187 5475 log.cc:824] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:35.227702 5477 tablet_bootstrap.cc:654] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:35.227744 5476 tablet_bootstrap.cc:654] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:35.228518 5476 log.cc:824] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:35.228518 5477 log.cc:824] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:35.228957 5475 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: No bootstrap required, opened a new log
I20260925 16:38:35.229151 5475 ts_tablet_manager.cc:1397] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:35.230438 5477 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: No bootstrap required, opened a new log
I20260925 16:38:35.230438 5476 tablet_bootstrap.cc:492] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: No bootstrap required, opened a new log
I20260925 16:38:35.230612 5476 ts_tablet_manager.cc:1397] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:35.230628 5477 ts_tablet_manager.cc:1397] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:35.232774 5475 raft_consensus.cc:348] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.233003 5475 raft_consensus.cc:374] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:35.233054 5475 raft_consensus.cc:729] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff8d09f43024bca944347250784a7e3, State: Initialized, Role: FOLLOWER
I20260925 16:38:35.233496 5475 consensus_queue.cc:260] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.234104 5458 heartbeater.cc:503] Master 127.0.97.254:44409 was elected leader, sending a full tablet report...
I20260925 16:38:35.234476 5475 ts_tablet_manager.cc:1428] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:38:35.235039 5476 raft_consensus.cc:348] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.235039 5477 raft_consensus.cc:348] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.235275 5476 raft_consensus.cc:374] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:35.235275 5477 raft_consensus.cc:374] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:35.235334 5477 raft_consensus.cc:729] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465d1759af5b4cca8f51bba58e7c5af5, State: Initialized, Role: FOLLOWER
I20260925 16:38:35.235329 5476 raft_consensus.cc:729] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99e8436d46b4e579c9af43734d6fcad, State: Initialized, Role: FOLLOWER
I20260925 16:38:35.235740 5477 consensus_queue.cc:260] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.235740 5476 consensus_queue.cc:260] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.236402 5196 heartbeater.cc:503] Master 127.0.97.254:44409 was elected leader, sending a full tablet report...
I20260925 16:38:35.236690 5327 heartbeater.cc:503] Master 127.0.97.254:44409 was elected leader, sending a full tablet report...
I20260925 16:38:35.237640 5477 ts_tablet_manager.cc:1428] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:38:35.237963 5476 ts_tablet_manager.cc:1428] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 16:38:35.247439 5328 tablet.cc:2369] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:35.260076 5482 raft_consensus.cc:482] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:35.260293 5482 raft_consensus.cc:504] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.261458 5482 leader_election.cc:288] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 465d1759af5b4cca8f51bba58e7c5af5 (127.0.97.194:38475), 6ff8d09f43024bca944347250784a7e3 (127.0.97.195:36469)
I20260925 16:38:35.263332 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.261676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38475 (local address 127.0.97.193:53089)
0925 16:38:35.261804 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.261821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.261895 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.262136 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.262156 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.262174 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.262746 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.262748 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.262760 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.262786 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.263053 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.263056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.263058 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.263157 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.263195 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 16:38:35.263334 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.261779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:53089 (local address 127.0.97.194:38475)
0925 16:38:35.261901 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.261905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.261920 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:35.261972 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.261975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.262009 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.262091 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.262922 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.262925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.262928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.262942 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.262967 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.262977 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.263065 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.263156 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.263195 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:38:35.263617 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.261686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:36469 (local address 127.0.97.193:47677)
0925 16:38:35.262298 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.262315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.262402 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.262647 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.262650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.262662 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:35.263033 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:38:35.263038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:35.263047 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:38:35.263069 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:35.263324 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:35.263327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:35.263330 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:38:35.263413 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:38:35.263448 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":282,"thread_start_us":139,"threads_started":1}
I20260925 16:38:35.263612 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.261826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:47677 (local address 127.0.97.195:36469)
0925 16:38:35.261960 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.261963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.262410 (+ 447us) server_negotiation.cc:407] Connection header received
0925 16:38:35.262490 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.262495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.262534 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.262615 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.263192 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.263195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.263198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.263209 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.263228 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.263238 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.263326 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.263436 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.263479 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:35.264039 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b9bac7191e4000b78a6548ceb57ba2" candidate_uuid: "d99e8436d46b4e579c9af43734d6fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465d1759af5b4cca8f51bba58e7c5af5" is_pre_election: true
I20260925 16:38:35.264195 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b9bac7191e4000b78a6548ceb57ba2" candidate_uuid: "d99e8436d46b4e579c9af43734d6fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff8d09f43024bca944347250784a7e3" is_pre_election: true
I20260925 16:38:35.264370 5281 raft_consensus.cc:2454] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99e8436d46b4e579c9af43734d6fcad in term 0.
I20260925 16:38:35.264488 5412 raft_consensus.cc:2454] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99e8436d46b4e579c9af43734d6fcad in term 0.
I20260925 16:38:35.264886 5083 leader_election.cc:302] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465d1759af5b4cca8f51bba58e7c5af5, d99e8436d46b4e579c9af43734d6fcad; no voters:
I20260925 16:38:35.265136 5482 raft_consensus.cc:2781] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:35.265214 5482 raft_consensus.cc:482] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:35.265256 5482 raft_consensus.cc:3037] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:35.266266 5482 raft_consensus.cc:504] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.266589 5482 leader_election.cc:288] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [CANDIDATE]: Term 1 election: Requested vote from peers 465d1759af5b4cca8f51bba58e7c5af5 (127.0.97.194:38475), 6ff8d09f43024bca944347250784a7e3 (127.0.97.195:36469)
I20260925 16:38:35.266908 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b9bac7191e4000b78a6548ceb57ba2" candidate_uuid: "d99e8436d46b4e579c9af43734d6fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465d1759af5b4cca8f51bba58e7c5af5"
I20260925 16:38:35.266960 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b9bac7191e4000b78a6548ceb57ba2" candidate_uuid: "d99e8436d46b4e579c9af43734d6fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff8d09f43024bca944347250784a7e3"
I20260925 16:38:35.267020 5281 raft_consensus.cc:3037] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:35.267086 5412 raft_consensus.cc:3037] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:35.268026 5281 raft_consensus.cc:2454] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99e8436d46b4e579c9af43734d6fcad in term 1.
I20260925 16:38:35.268144 5412 raft_consensus.cc:2454] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99e8436d46b4e579c9af43734d6fcad in term 1.
I20260925 16:38:35.268376 5083 leader_election.cc:302] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465d1759af5b4cca8f51bba58e7c5af5, d99e8436d46b4e579c9af43734d6fcad; no voters:
I20260925 16:38:35.268545 5482 raft_consensus.cc:2781] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:35.268793 5482 raft_consensus.cc:686] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 1 LEADER]: Becoming Leader. State: Replica: d99e8436d46b4e579c9af43734d6fcad, State: Running, Role: LEADER
I20260925 16:38:35.269145 5482 consensus_queue.cc:237] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } }
I20260925 16:38:35.272286 5007 catalog_manager.cc:5495] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad reported cstate change: term changed from 0 to 1, leader changed from <none> to d99e8436d46b4e579c9af43734d6fcad (127.0.97.193). New cstate: current_term: 1 leader_uuid: "d99e8436d46b4e579c9af43734d6fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99e8436d46b4e579c9af43734d6fcad" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:35.292028 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.290486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.97.193:40977)
0925 16:38:35.290633 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.290636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.290648 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:35.290717 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.290719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.290774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.290874 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.291622 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.291626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.291629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.291647 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.291665 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.291672 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.291764 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.291847 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.291886 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:38:35.298023 5412 raft_consensus.cc:1261] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 1 FOLLOWER]: Refusing update from remote peer d99e8436d46b4e579c9af43734d6fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:35.298712 5281 raft_consensus.cc:1261] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 1 FOLLOWER]: Refusing update from remote peer d99e8436d46b4e579c9af43734d6fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:35.298743 5482 consensus_queue.cc:1035] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff8d09f43024bca944347250784a7e3" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:35.299364 5482 consensus_queue.cc:1035] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "465d1759af5b4cca8f51bba58e7c5af5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:35.307161 5489 mvcc.cc:205] Tried to move back new op lower bound from 7333291275453222912 to 7333291275345309696. Current Snapshot: MvccSnapshot[applied={T|T < 7333291275453222912}]
I20260925 16:38:35.315912 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:35.316092 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:35.318620 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a303c8cb1104f368e4106c914fbca80: Sending DeleteTablet for 3 replicas of tablet f3b9bac7191e4000b78a6548ceb57ba2
I20260925 16:38:35.319373 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:35 UTC) from {username='slave'} at 127.0.0.1:41518
I20260925 16:38:35.319388 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:35 UTC) from {username='slave'} at 127.0.0.1:56368
I20260925 16:38:35.319883 5498 tablet_replica.cc:330] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad: stopping tablet replica
I20260925 16:38:35.319891 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b9bac7191e4000b78a6548ceb57ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:35 UTC) from {username='slave'} at 127.0.0.1:35546
I20260925 16:38:35.320178 5499 tablet_replica.cc:330] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: stopping tablet replica
I20260925 16:38:35.320319 5500 tablet_replica.cc:330] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: stopping tablet replica
I20260925 16:38:35.320605 5498 raft_consensus.cc:2229] T f3b9bac7191e4000b78a6548ceb57ba2 P d99e8436d46b4e579c9af43734d6fcad [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:35.320604 5500 raft_consensus.cc:2229] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:35.320690 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5068
I20260925 16:38:35.320924 5500 raft_consensus.cc:2258] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:35.321878 5499 raft_consensus.cc:2229] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:35.322244 5499 raft_consensus.cc:2258] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:35.322630 5500 ts_tablet_manager.cc:1905] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:35.323897 5499 ts_tablet_manager.cc:1905] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:35.325376 5500 ts_tablet_manager.cc:1918] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:35.325482 5500 log.cc:1196] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/f3b9bac7191e4000b78a6548ceb57ba2
I20260925 16:38:35.325814 5500 ts_tablet_manager.cc:1939] T f3b9bac7191e4000b78a6548ceb57ba2 P 6ff8d09f43024bca944347250784a7e3: Deleting consensus metadata
I20260925 16:38:35.326655 5499 ts_tablet_manager.cc:1918] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:35.326750 5499 log.cc:1196] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/f3b9bac7191e4000b78a6548ceb57ba2
I20260925 16:38:35.327044 5499 ts_tablet_manager.cc:1939] T f3b9bac7191e4000b78a6548ceb57ba2 P 465d1759af5b4cca8f51bba58e7c5af5: Deleting consensus metadata
I20260925 16:38:35.327330 4993 catalog_manager.cc:4841] TS 6ff8d09f43024bca944347250784a7e3 (127.0.97.195:36469): tablet f3b9bac7191e4000b78a6548ceb57ba2 (table test-table [id=f1322b5e8d6d4e9db592343a0d2eb2bf]) successfully deleted
W20260925 16:38:35.328080 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:40977 (error 108)
I20260925 16:38:35.328166 4992 catalog_manager.cc:4841] TS 465d1759af5b4cca8f51bba58e7c5af5 (127.0.97.194:38475): tablet f3b9bac7191e4000b78a6548ceb57ba2 (table test-table [id=f1322b5e8d6d4e9db592343a0d2eb2bf]) successfully deleted
I20260925 16:38:35.328223 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5199
I20260925 16:38:35.329797 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.329444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40977 (local address 127.0.0.1:37776)
0925 16:38:35.329604 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.329685 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:38:35.329977 4994 catalog_manager.cc:4584] TS d99e8436d46b4e579c9af43734d6fcad (127.0.97.193:40977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3b9bac7191e4000b78a6548ceb57ba2: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40977: connect: Connection refused (error 111)
I20260925 16:38:35.334153 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5330
I20260925 16:38:35.339589 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 4977
2026-09-25T16:38:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (954 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 16:38:35.360841 391 mini_oidc.cc:152] Starting JWKS server
I20260925 16:38:35.362421 391 webserver.cc:460] Webserver started at https://127.0.0.1:36785/ using document root <none> and password file <none>
I20260925 16:38:35.362550 391 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:38:35.362764 391 webserver.cc:460] Webserver started at http://127.0.0.1:39545/ using document root <none> and password file <none>
2026-09-25T16:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:35Z Disabled control of system clock
I20260925 16:38:35.371173 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42703
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42703
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/testchainca.cert with env {}
W20260925 16:38:35.480052 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:35.480367 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:35.480408 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:35.484364 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:35.484424 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:35.484439 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:35.484457 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:35.484472 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:38:35.484488 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:38:35.484501 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36785/jwks.json
I20260925 16:38:35.489959 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:35.490954 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:35.491896 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:35.500164 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:35.500198 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:35.500192 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:35.500630 5507 server_base.cc:1034] running on GCE node
I20260925 16:38:35.501159 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:35.502318 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:35.503501 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790354315503481 us; error 57 us; skew 500 ppm
I20260925 16:38:35.505327 5507 webserver.cc:460] Webserver started at http://127.0.97.254:44857/ using document root <none> and password file <none>
I20260925 16:38:35.505903 5507 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:35.505995 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:35.506214 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:35.508179 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "5b1d5aaa704a4b00a3a90d89391a8d60"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9c14a5a24d8bdfc83392feb4b1c10814"
server_key_iv: "3659488377858a76ca62eec9a8f40066"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.508754 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "5b1d5aaa704a4b00a3a90d89391a8d60"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9c14a5a24d8bdfc83392feb4b1c10814"
server_key_iv: "3659488377858a76ca62eec9a8f40066"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.512422 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:35.514951 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:35.516137 5507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:38:35.516283 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "5b1d5aaa704a4b00a3a90d89391a8d60"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "9c14a5a24d8bdfc83392feb4b1c10814"
server_key_iv: "3659488377858a76ca62eec9a8f40066"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.516395 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:35.535439 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:35.536119 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:35.536332 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:35.543089 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42703
I20260925 16:38:35.543100 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42703 every 8 connection(s)
I20260925 16:38:35.543990 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:35.546355 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5507
I20260925 16:38:35.546470 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:35.546751 391 external_mini_cluster.cc:1428] Setting key b63e8f8867a1f5e219b8d49e9beb223e
I20260925 16:38:35.547200 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:35.553932 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60: Bootstrap starting.
I20260925 16:38:35.555629 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.548027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.97.254:42703)
0925 16:38:35.548619 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.548645 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.548679 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:38:35.549418 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.549436 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.549812 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.550077 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.550408 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.551324 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.552501 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.552950 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.553060 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.553910 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.553915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.553926 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.554366 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.554417 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.554438 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.554511 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.554673 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.554962 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260925 16:38:35.556185 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:35.556900 5574 log.cc:824] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:35.558398 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:38:35.558941 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60: No bootstrap required, opened a new log
I20260925 16:38:35.562529 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } }
I20260925 16:38:35.562726 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:35.562805 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1d5aaa704a4b00a3a90d89391a8d60, State: Initialized, Role: FOLLOWER
I20260925 16:38:35.563202 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } }
I20260925 16:38:35.563373 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:35.563457 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:35.563593 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:35.564481 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } }
I20260925 16:38:35.564792 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b1d5aaa704a4b00a3a90d89391a8d60; no voters:
I20260925 16:38:35.565084 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:35.565207 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:35.565469 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1d5aaa704a4b00a3a90d89391a8d60, State: Running, Role: LEADER
I20260925 16:38:35.565766 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } }
I20260925 16:38:35.566314 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:35.567323 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b1d5aaa704a4b00a3a90d89391a8d60. Latest consensus state: current_term: 1 leader_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } } }
I20260925 16:38:35.567461 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:35.567728 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b1d5aaa704a4b00a3a90d89391a8d60" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42703 } } }
I20260925 16:38:35.567817 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:35.568022 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:35.570883 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:35.576436 5588 catalog_manager.cc:1347] Generated new cluster ID: f7ce4e44935646dea993b0f598fd9c94
I20260925 16:38:35.576501 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:35.604039 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:35.604986 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:35.613054 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b1d5aaa704a4b00a3a90d89391a8d60: Generated new TSK 0
I20260925 16:38:35.613777 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:38:35.647667 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 16:38:35.647789 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:38:35.648377 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.631434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.97.254:42703)
0925 16:38:35.631570 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.631574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.631744 (+ 170us) server_negotiation.cc:407] Connection header received
0925 16:38:35.631935 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.631939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.631993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.632111 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:38:35.632343 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.633136 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.634022 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.634375 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.634479 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.634763 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:38:35.647962 (+ 13199us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:38:35.648228 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 16:38:35.648458 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 16:38:35.649610 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:35.631281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42703 (local address 127.0.0.1:45274)
0925 16:38:35.631663 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.631676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.631778 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.632125 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.632128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.632133 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:38:35.632236 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:35.632240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.633255 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.633257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:35.633911 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:35.633920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.634486 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.634489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:35.634551 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.634592 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:38:35.648174 (+ 13582us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:38:35.648365 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.254:42703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.254:42703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 16:38:35.650482 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5507
2026-09-25T16:38:35Z chronyd exiting
I20260925 16:38:35.687237 391 test_util.cc:175] -----------------------------------------------
I20260925 16:38:35.687333 391 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790354288967115-391-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 16:38:35.688375 391 mini_oidc.cc:152] Starting JWKS server
I20260925 16:38:35.689195 391 webserver.cc:460] Webserver started at https://127.0.0.1:35241/ using document root <none> and password file <none>
I20260925 16:38:35.689332 391 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:38:35.689605 391 webserver.cc:460] Webserver started at http://127.0.0.1:40439/ using document root <none> and password file <none>
2026-09-25T16:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:35Z Disabled control of system clock
I20260925 16:38:35.697922 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44549
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35241/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/testchainca.cert with env {}
W20260925 16:38:35.805305 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:35.805589 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:35.805668 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:35.809644 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:35.809736 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:35.809764 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:35.809804 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:35.809835 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:38:35.809870 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:38:35.809897 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35241/jwks.json
I20260925 16:38:35.815205 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35241/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:35.816217 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:35.817195 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:35.825491 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:35.825497 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:35.825729 5610 server_base.cc:1034] running on GCE node
W20260925 16:38:35.825546 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:35.826345 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:35.827406 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:35.828567 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790354315828537 us; error 42 us; skew 500 ppm
I20260925 16:38:35.830376 5610 webserver.cc:460] Webserver started at http://127.0.97.254:46645/ using document root <none> and password file <none>
I20260925 16:38:35.830915 5610 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:35.830973 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:35.831214 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:35.833153 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "bda1497c5844430ba899b609e1fde38b"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "94d5d969278ed4ed46da68eb37d5058f"
server_key_iv: "b72d6393c8853f266d640f0c50182127"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.833755 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "bda1497c5844430ba899b609e1fde38b"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "94d5d969278ed4ed46da68eb37d5058f"
server_key_iv: "b72d6393c8853f266d640f0c50182127"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.837198 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:35.839666 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:35.840955 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:35.841107 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "bda1497c5844430ba899b609e1fde38b"
format_stamp: "Formatted at 2026-09-25 16:38:35 on dist-test-slave-kxl8"
server_key: "94d5d969278ed4ed46da68eb37d5058f"
server_key_iv: "b72d6393c8853f266d640f0c50182127"
server_key_version: "encryptionkey@0"
I20260925 16:38:35.841218 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:35.896516 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:35.897240 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:35.897500 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:35.904333 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44549
I20260925 16:38:35.904351 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44549 every 8 connection(s)
I20260925 16:38:35.905328 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:35.908291 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:35.913594 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b: Bootstrap starting.
I20260925 16:38:35.913628 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5610
I20260925 16:38:35.913748 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:35.914021 391 external_mini_cluster.cc:1428] Setting key befff3430da4fec76cf042c11dff2fa5
I20260925 16:38:35.916115 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:35.916862 5677 log.cc:824] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:35.918641 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b: No bootstrap required, opened a new log
I20260925 16:38:35.923162 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } }
I20260925 16:38:35.923420 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.915241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.0.97.254:44549)
0925 16:38:35.915864 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.915878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.915913 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:38:35.916563 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.916581 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.916967 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.917363 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:35.917775 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.918701 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.919664 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.920388 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.920567 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.921340 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:35.921346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:35.921361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:35.921860 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:35.921930 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:35.921962 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:35.922084 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:35.922202 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 16:38:35.922495 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260925 16:38:35.923475 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:35.923518 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda1497c5844430ba899b609e1fde38b, State: Initialized, Role: FOLLOWER
I20260925 16:38:35.924633 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } }
I20260925 16:38:35.924762 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:35.924849 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:35.924968 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:35.926019 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } }
I20260925 16:38:35.926081 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:38:35.926357 5677 leader_election.cc:302] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda1497c5844430ba899b609e1fde38b; no voters:
I20260925 16:38:35.926635 5677 leader_election.cc:288] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:35.926765 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:35.926992 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [term 1 LEADER]: Becoming Leader. State: Replica: bda1497c5844430ba899b609e1fde38b, State: Running, Role: LEADER
I20260925 16:38:35.927279 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } }
I20260925 16:38:35.927878 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:35.928445 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda1497c5844430ba899b609e1fde38b. Latest consensus state: current_term: 1 leader_uuid: "bda1497c5844430ba899b609e1fde38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } } }
I20260925 16:38:35.928570 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:35.928814 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda1497c5844430ba899b609e1fde38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1497c5844430ba899b609e1fde38b" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44549 } } }
I20260925 16:38:35.928923 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:35.928974 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:35.931807 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:35.937664 5688 catalog_manager.cc:1347] Generated new cluster ID: 771ec3584c054a938827d587217ddae9
I20260925 16:38:35.937745 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:35.954818 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:35.955986 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:35.962795 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda1497c5844430ba899b609e1fde38b: Generated new TSK 0
I20260925 16:38:35.963500 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:38:35.993420 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 16:38:35.993522 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:38:35.994285 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:35.977646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.97.254:44549)
0925 16:38:35.977780 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:35.977785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:35.977937 (+ 152us) server_negotiation.cc:407] Connection header received
0925 16:38:35.978066 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:35.978069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:35.978125 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:35.978249 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:38:35.978445 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.979277 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.980104 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.980459 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.980560 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.980821 (+ 261us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:38:35.993827 (+ 13006us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:38:35.994118 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 16:38:35.994357 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 16:38:35.994382 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:35.977513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44549 (local address 127.0.0.1:37858)
0925 16:38:35.977854 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:35.977866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:35.977960 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:35.978252 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:35.978254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:35.978259 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:38:35.978349 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:35.978353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.979382 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.979385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:35.979996 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:35.980004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:35.980569 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:35.980572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:35.980638 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:35.980659 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:38:35.994052 (+ 13393us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:38:35.994213 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.254:44549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260925 16:38:35.995329 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5610
2026-09-25T16:38:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (346 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 16:38:36.034773 391 mini_oidc.cc:152] Starting JWKS server
I20260925 16:38:36.035627 391 webserver.cc:460] Webserver started at https://127.0.0.1:45501/ using document root <none> and password file <none>
I20260925 16:38:36.035741 391 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:38:36.036031 391 webserver.cc:460] Webserver started at http://127.0.0.1:45515/ using document root <none> and password file <none>
2026-09-25T16:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:36Z Disabled control of system clock
I20260925 16:38:36.044379 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42197
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42197
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:38:36.152495 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:36.152788 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:36.152861 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:36.156903 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:36.156998 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:36.157029 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:36.157070 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:36.157105 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:38:36.157142 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:38:36.157174 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45501/jwks.json
I20260925 16:38:36.161949 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:36.163005 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:36.164388 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:36.172691 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:36.172834 5713 server_base.cc:1034] running on GCE node
W20260925 16:38:36.172691 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:36.172706 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:36.173504 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:36.174567 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:36.175748 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790354316175727 us; error 53 us; skew 500 ppm
I20260925 16:38:36.177611 5713 webserver.cc:460] Webserver started at http://127.0.97.254:43031/ using document root <none> and password file <none>
I20260925 16:38:36.178169 5713 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:36.178257 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:36.178472 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:36.180485 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "fb36e760be5b46c4b3af3047b61e12f9"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "95d57c7a2c568a87626290069cedb175"
server_key_iv: "532e19cffae5a6e44072541af9db16bb"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.181033 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "fb36e760be5b46c4b3af3047b61e12f9"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "95d57c7a2c568a87626290069cedb175"
server_key_iv: "532e19cffae5a6e44072541af9db16bb"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.184502 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:36.186859 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:36.188017 5713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:36.188148 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "fb36e760be5b46c4b3af3047b61e12f9"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "95d57c7a2c568a87626290069cedb175"
server_key_iv: "532e19cffae5a6e44072541af9db16bb"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.188257 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:36.215533 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:36.216327 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:36.216536 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:36.223722 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42197
I20260925 16:38:36.223773 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42197 every 8 connection(s)
I20260925 16:38:36.224745 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:36.227859 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:36.229728 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5713
I20260925 16:38:36.229849 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:36.230106 391 external_mini_cluster.cc:1428] Setting key bfff5650067ca0ad4848ba2cb6c79b5f
I20260925 16:38:36.234498 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9: Bootstrap starting.
I20260925 16:38:36.237000 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:36.237866 5780 log.cc:824] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:36.239497 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:36.231438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.0.97.254:42197)
0925 16:38:36.232075 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:36.232093 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:36.232130 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:38:36.232824 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:36.232842 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:36.233241 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:36.233603 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:36.233960 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.235022 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.236189 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.236750 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.236875 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.237826 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:36.237831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:36.237844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:36.238269 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:36.238323 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:36.238343 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:36.238457 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:36.238583 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 16:38:36.238828 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260925 16:38:36.239851 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9: No bootstrap required, opened a new log
I20260925 16:38:36.242369 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:38:36.244279 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } }
I20260925 16:38:36.244527 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:36.244590 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb36e760be5b46c4b3af3047b61e12f9, State: Initialized, Role: FOLLOWER
I20260925 16:38:36.244992 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } }
I20260925 16:38:36.245139 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:36.245236 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:36.245369 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:36.246304 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } }
I20260925 16:38:36.246649 5780 leader_election.cc:302] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb36e760be5b46c4b3af3047b61e12f9; no voters:
I20260925 16:38:36.246963 5780 leader_election.cc:288] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:36.247082 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:36.247319 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [term 1 LEADER]: Becoming Leader. State: Replica: fb36e760be5b46c4b3af3047b61e12f9, State: Running, Role: LEADER
I20260925 16:38:36.247718 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } }
I20260925 16:38:36.248255 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:36.249544 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb36e760be5b46c4b3af3047b61e12f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } } }
I20260925 16:38:36.249670 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:36.250016 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb36e760be5b46c4b3af3047b61e12f9. Latest consensus state: current_term: 1 leader_uuid: "fb36e760be5b46c4b3af3047b61e12f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb36e760be5b46c4b3af3047b61e12f9" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42197 } } }
I20260925 16:38:36.250120 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:36.250144 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:36.253154 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:36.259099 5794 catalog_manager.cc:1347] Generated new cluster ID: 990ec916ad43477f879928cd171da34b
I20260925 16:38:36.259173 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:36.268150 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:36.269388 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:36.279279 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb36e760be5b46c4b3af3047b61e12f9: Generated new TSK 0
I20260925 16:38:36.279975 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:38:36.300719 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:36.295863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.97.254:42197)
0925 16:38:36.295998 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:36.296005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:36.296158 (+ 153us) server_negotiation.cc:407] Connection header received
0925 16:38:36.296326 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:36.296328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:36.296373 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:36.296454 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:38:36.296687 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.297216 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.298448 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.298661 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.298730 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.299037 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:38:36.300358 (+ 1321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:38:36.300584 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 16:38:36.300781 5783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 16:38:36.302634 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:36.295778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42197 (local address 127.0.0.1:52762)
0925 16:38:36.296069 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:36.296084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:36.296194 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:36.296474 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:36.296477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:36.296485 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:38:36.296591 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:36.296597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.297342 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.297346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:36.298330 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:36.298339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.298770 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.298774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:36.298860 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.298888 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:38:36.300527 (+ 1639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:38:36.302462 (+ 1935us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.254:42197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":1680,"thread_start_us":112,"threads_started":1}
I20260925 16:38:36.303169 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5713
E20260925 16:38:36.311651 5810 webserver.cc:533] Webserver: sslize failed:
E20260925 16:38:36.311735 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T16:38:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T16:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:36Z Disabled control of system clock
I20260925 16:38:36.349499 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:43357
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:43357
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/testchainca.cert with env {}
W20260925 16:38:36.457541 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:36.457818 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:36.457857 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:36.461930 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:36.461990 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:36.462007 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:36.462025 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:36.462040 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:38:36.462056 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:38:36.462076 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 16:38:36.467645 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:43357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:36.468699 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:36.469733 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:36.478206 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:36.478365 5814 server_base.cc:1034] running on GCE node
W20260925 16:38:36.478245 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:36.478587 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:36.479152 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:36.480274 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:36.481464 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790354316481443 us; error 32 us; skew 500 ppm
I20260925 16:38:36.483434 5814 webserver.cc:460] Webserver started at http://127.0.97.254:41847/ using document root <none> and password file <none>
I20260925 16:38:36.484050 5814 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:36.484148 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:36.484382 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:36.486557 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "b15a843a7bd649629072924d82d9040f"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "dc3320b97b6ae36b4e1e4fddd7e4ab70"
server_key_iv: "c9f0752b1fc0e985bdfb2658f6df555a"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.487134 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "b15a843a7bd649629072924d82d9040f"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "dc3320b97b6ae36b4e1e4fddd7e4ab70"
server_key_iv: "c9f0752b1fc0e985bdfb2658f6df555a"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.490887 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:38:36.493438 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:36.494628 5814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:36.494786 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "b15a843a7bd649629072924d82d9040f"
format_stamp: "Formatted at 2026-09-25 16:38:36 on dist-test-slave-kxl8"
server_key: "dc3320b97b6ae36b4e1e4fddd7e4ab70"
server_key_iv: "c9f0752b1fc0e985bdfb2658f6df555a"
server_key_version: "encryptionkey@0"
I20260925 16:38:36.494910 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:36.509254 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:36.510205 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:36.510411 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:36.517380 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:43357
I20260925 16:38:36.517385 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:43357 every 8 connection(s)
I20260925 16:38:36.518325 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:36.521448 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:36.524879 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5814
I20260925 16:38:36.525003 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:36.525324 391 external_mini_cluster.cc:1428] Setting key f6190a935140c941643465f7fdce815a
I20260925 16:38:36.527240 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f: Bootstrap starting.
I20260925 16:38:36.529395 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:36.530099 5881 log.cc:824] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:36.531855 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f: No bootstrap required, opened a new log
I20260925 16:38:36.534235 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:36.526638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.97.254:43357)
0925 16:38:36.527242 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:36.527256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:36.527326 (+ 70us) server_negotiation.cc:407] Connection header received
0925 16:38:36.527956 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:36.527973 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:36.528360 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:36.528702 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:36.529123 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.530219 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.531131 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.531601 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.531771 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.532616 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:36.532621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:36.532634 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:38:36.533003 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:36.533056 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:38:36.533075 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:36.533183 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:36.533331 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:38:36.533574 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260925 16:38:36.535475 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } }
I20260925 16:38:36.535727 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:36.535774 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a843a7bd649629072924d82d9040f, State: Initialized, Role: FOLLOWER
I20260925 16:38:36.536206 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } }
I20260925 16:38:36.536336 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:36.536423 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:36.536531 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:36.537014 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:38:36.537652 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } }
I20260925 16:38:36.538002 5881 leader_election.cc:302] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15a843a7bd649629072924d82d9040f; no voters:
I20260925 16:38:36.538290 5881 leader_election.cc:288] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:36.538419 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:36.538736 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [term 1 LEADER]: Becoming Leader. State: Replica: b15a843a7bd649629072924d82d9040f, State: Running, Role: LEADER
I20260925 16:38:36.539034 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } }
I20260925 16:38:36.539469 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:36.540674 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15a843a7bd649629072924d82d9040f. Latest consensus state: current_term: 1 leader_uuid: "b15a843a7bd649629072924d82d9040f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } } }
I20260925 16:38:36.540793 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:36.541157 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:36.541098 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15a843a7bd649629072924d82d9040f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15a843a7bd649629072924d82d9040f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 43357 } } }
I20260925 16:38:36.541314 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:36.543921 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:36.549188 5895 catalog_manager.cc:1347] Generated new cluster ID: 529a1626b46543c9b62e8ed0ae2b890e
I20260925 16:38:36.549244 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:36.561156 5895 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:36.562098 5895 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:36.566593 5895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15a843a7bd649629072924d82d9040f: Generated new TSK 0
I20260925 16:38:36.567245 5895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:38:37.004885 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:36.575927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.97.254:43357)
0925 16:38:36.576075 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:36.576079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:36.576210 (+ 131us) server_negotiation.cc:407] Connection header received
0925 16:38:36.576413 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:36.576416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:36.576464 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:36.576578 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:38:36.576858 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.577484 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.578349 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.578576 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.578689 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.578994 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:38:37.004401 (+425407us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:38:37.004722 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 16:38:37.004978 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 16:38:37.004995 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:36.575802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:43357 (local address 127.0.0.1:52688)
0925 16:38:36.576103 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:36.576125 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:36.576270 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:36.576575 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:36.576578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:36.576586 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:38:36.576744 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:36.576750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.577595 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.577599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:36.578227 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:36.578235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:36.578710 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:36.578714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:36.578797 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:36.578818 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:38:37.004632 (+425814us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:38:37.004822 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.254:43357: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
I20260925 16:38:37.005771 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5814
2026-09-25T16:38:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (683 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:37 dist-test-slave-kxl8 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:37 dist-test-slave-kxl8 krb5kdc[5922](info): set up 2 sockets
Sep 25 16:38:37 dist-test-slave-kxl8 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260925 16:38:38.281708 391 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:38 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:38.400609 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:35333
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:35333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:38.514863 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:38.515128 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:38.515169 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:38.519235 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:38.519295 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:38.519315 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:38.519331 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:38.519369 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:38.524816 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:35333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:38.525988 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:38.527009 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:38.535390 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:38.535490 5935 server_base.cc:1034] running on GCE node
W20260925 16:38:38.535396 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:38.535390 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:38.536185 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:38.537387 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:38.538551 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790354318538535 us; error 32 us; skew 500 ppm
Sep 25 16:38:38 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:38.541251 5935 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:38.542116 5935 webserver.cc:460] Webserver started at http://127.0.97.254:41263/ using document root <none> and password file <none>
I20260925 16:38:38.542618 5935 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:38.542676 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:38.542842 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:38.544791 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "1294f8d884a74699940e02e5b7e4d825"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "7226e23b6a1f1707746ea7900103ab59"
server_key_iv: "24d117f60aadc1b796c446f175b21b16"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.545377 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "1294f8d884a74699940e02e5b7e4d825"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "7226e23b6a1f1707746ea7900103ab59"
server_key_iv: "24d117f60aadc1b796c446f175b21b16"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.548903 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:38.551424 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:38.552739 5935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:38.552847 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "1294f8d884a74699940e02e5b7e4d825"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "7226e23b6a1f1707746ea7900103ab59"
server_key_iv: "24d117f60aadc1b796c446f175b21b16"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.552930 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:38.565496 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:38.568684 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:38.568838 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:38.575517 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:35333
I20260925 16:38:38.575793 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:35333 every 8 connection(s)
I20260925 16:38:38.576507 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:38.579248 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:38.584513 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: Bootstrap starting.
I20260925 16:38:38.586352 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 5935
I20260925 16:38:38.586449 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:38.586846 391 external_mini_cluster.cc:1428] Setting key 580cc81140353d2d5e448dba2b298173
I20260925 16:38:38.587137 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:38.588056 6003 log.cc:824] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:38.590131 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: No bootstrap required, opened a new log
I20260925 16:38:38.594290 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } }
I20260925 16:38:38.594578 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:38.594635 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1294f8d884a74699940e02e5b7e4d825, State: Initialized, Role: FOLLOWER
I20260925 16:38:38.595168 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } }
I20260925 16:38:38.595358 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:38.595431 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:38.595578 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:38:38 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:38.596532 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } }
I20260925 16:38:38.596868 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1294f8d884a74699940e02e5b7e4d825; no voters:
I20260925 16:38:38.597194 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:38.597265 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:38.597502 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [term 1 LEADER]: Becoming Leader. State: Replica: 1294f8d884a74699940e02e5b7e4d825, State: Running, Role: LEADER
I20260925 16:38:38.597834 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } }
I20260925 16:38:38.598486 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:38.601325 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1294f8d884a74699940e02e5b7e4d825. Latest consensus state: current_term: 1 leader_uuid: "1294f8d884a74699940e02e5b7e4d825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } } }
I20260925 16:38:38.601265 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1294f8d884a74699940e02e5b7e4d825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1294f8d884a74699940e02e5b7e4d825" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 35333 } } }
I20260925 16:38:38.601432 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:38.602092 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:38.587969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.97.254:35333)
0925 16:38:38.588378 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:38.588414 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:38.588462 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:38:38.589226 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:38.589245 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:38.589624 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:38.589915 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:38.590815 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.591772 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.592978 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.593500 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.593617 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:38.596506 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:38.596511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:38.596525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:38.596550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:38.598573 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:38.599067 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:38.599072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:38.599078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:38.599221 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:38.599553 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:38.599557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:38.599558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:38.600036 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:38.600230 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:38.600620 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 16:38:38.600972 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":135,"threads_started":1}
I20260925 16:38:38.601432 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825 [sys.catalog]: This master's current role is: LEADER
W20260925 16:38:38.603897 6023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:38:38.603984 6023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:38:38.604086 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:38.604875 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:38.610522 6024 catalog_manager.cc:1347] Generated new cluster ID: 6711618bab9342f9b238687f23ea9640
I20260925 16:38:38.610598 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:38.618664 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:38.619503 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:38.627468 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1294f8d884a74699940e02e5b7e4d825: Generated new TSK 0
I20260925 16:38:38.628258 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:38.664296 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:38.774024 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:38.774336 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:38.774410 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:38.778440 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:38.778530 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:38.778620 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:38.783838 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:38.784870 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:38.785895 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:38.795120 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:38.795213 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:38.795120 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:38.795315 6029 server_base.cc:1034] running on GCE node
I20260925 16:38:38.795851 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:38.796473 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:38.797652 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790354318797637 us; error 29 us; skew 500 ppm
Sep 25 16:38:38 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:38.800591 6029 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:38.801446 6029 webserver.cc:460] Webserver started at http://127.0.97.193:40233/ using document root <none> and password file <none>
I20260925 16:38:38.801965 6029 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:38.802022 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:38.802191 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:38.804244 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "aa1b622cbd2445fca9d69d7c8970d754"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "d6caa5c3033cafca9c8a9d8dc955d945"
server_key_iv: "37cb2c747fab1f5c610eafb89299adb7"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.804800 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "aa1b622cbd2445fca9d69d7c8970d754"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "d6caa5c3033cafca9c8a9d8dc955d945"
server_key_iv: "37cb2c747fab1f5c610eafb89299adb7"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.808326 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:38:38.810688 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:38.811852 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:38.811985 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "aa1b622cbd2445fca9d69d7c8970d754"
format_stamp: "Formatted at 2026-09-25 16:38:38 on dist-test-slave-kxl8"
server_key: "d6caa5c3033cafca9c8a9d8dc955d945"
server_key_iv: "37cb2c747fab1f5c610eafb89299adb7"
server_key_version: "encryptionkey@0"
I20260925 16:38:38.812088 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:38.857304 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:38.860570 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:38.860785 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:38.861402 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:38.862246 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:38.862313 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:38.862376 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:38.862401 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:38.870959 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:43493
I20260925 16:38:38.870990 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:43493 every 8 connection(s)
I20260925 16:38:38.871874 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
Sep 25 16:38:38 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:38.881387 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6029
I20260925 16:38:38.881521 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:38.881793 391 external_mini_cluster.cc:1428] Setting key fce08fe9291685e0b6a0b7a7e37ff36f
I20260925 16:38:38.884805 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:38.873637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:35879 (local address 127.0.97.254:35333)
0925 16:38:38.873792 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:38.873797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:38.874652 (+ 855us) server_negotiation.cc:407] Connection header received
0925 16:38:38.875958 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:38.875961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:38.876005 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:38.876087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:38.877148 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.877752 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.878810 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.879023 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.879098 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:38.881318 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:38.881322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:38.881325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:38.881353 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:38.882822 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:38.883335 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:38.883340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:38.883342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:38.883426 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:38.883760 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:38.883763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:38.883765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:38.883938 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:38.884044 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:38.884508 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 16:38:38.884642 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:38:38.885620 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:38.874077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:35333 (local address 127.0.97.193:35879)
0925 16:38:38.874506 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:38.874547 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:38.875750 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:38.876240 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:38.876247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:38.876580 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:38.877020 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:38.877032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.877862 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.877865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:38.878711 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:38.878719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:38.879141 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:38.879143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:38.879276 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:38.879780 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:38.879794 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:38.881133 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:38.882960 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:38.882965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:38.882978 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:38.883228 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:38.883520 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:38.883523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:38.883525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:38.883659 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:38.884053 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:38.884063 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:38.884287 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 16:38:38.884469 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 16:38:38.884766 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260925 16:38:38.886790 6158 heartbeater.cc:347] Connected to a master server at 127.0.97.254:35333
I20260925 16:38:38.887053 6158 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:38.887630 6158 heartbeater.cc:511] Master 127.0.97.254:35333 requested a full tablet report, sending...
I20260925 16:38:38.889429 5967 ts_manager.cc:194] Registered new tserver with Master: aa1b622cbd2445fca9d69d7c8970d754 (127.0.97.193:43493)
I20260925 16:38:38.891048 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:35879
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:38.908357 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:39.021144 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:39.021441 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:39.021519 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:39.025640 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:39.025700 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:39.025800 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:39.031145 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:39.032276 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:39.033329 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:39.043345 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:39.043352 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:39.043447 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:39.043653 6163 server_base.cc:1034] running on GCE node
I20260925 16:38:39.044068 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:39.044679 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:39.045857 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790354319045823 us; error 47 us; skew 500 ppm
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.048527 6163 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:39.049433 6163 webserver.cc:460] Webserver started at http://127.0.97.194:33309/ using document root <none> and password file <none>
I20260925 16:38:39.049947 6163 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:39.050002 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:39.050163 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:39.052121 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "c8bf899d5a194184985cbfdd3bb279fd"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "b52eace9633047174ce7f376a293f32f"
server_key_iv: "06cfcb218d35a0112e6d1d8c6ecb22da"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.052631 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "c8bf899d5a194184985cbfdd3bb279fd"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "b52eace9633047174ce7f376a293f32f"
server_key_iv: "06cfcb218d35a0112e6d1d8c6ecb22da"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.056380 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 16:38:39.058856 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.060009 6163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:39.060144 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "c8bf899d5a194184985cbfdd3bb279fd"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "b52eace9633047174ce7f376a293f32f"
server_key_iv: "06cfcb218d35a0112e6d1d8c6ecb22da"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.060256 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:39.079460 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:39.083336 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:39.083559 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:39.084185 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:39.085049 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:39.085122 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.085187 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:39.085239 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.094374 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:42225
I20260925 16:38:39.094393 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:42225 every 8 connection(s)
I20260925 16:38:39.095329 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:39.104941 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6163
I20260925 16:38:39.105069 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:39.105449 391 external_mini_cluster.cc:1428] Setting key 9f0486c3491a6d3d66cdd95c88b9d905
I20260925 16:38:39.110183 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.097176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:32943 (local address 127.0.97.254:35333)
0925 16:38:39.097344 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.097349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.098167 (+ 818us) server_negotiation.cc:407] Connection header received
0925 16:38:39.099735 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.099738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.099806 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.099911 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.101436 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.101971 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.103122 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.103343 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.103455 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.106005 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:39.106009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:39.106012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:39.106039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:39.107808 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.108294 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.108300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.108305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.108408 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.108856 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.108858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.108859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.109026 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:39.109100 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:39.109792 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 16:38:39.109946 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:38:39.110941 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.097600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:35333 (local address 127.0.97.194:32943)
0925 16:38:39.098018 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:39.098060 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:39.099527 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:39.100081 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:39.100088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:39.100595 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:39.101240 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:39.101256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.102117 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.102125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:39.102996 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:39.103005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.103496 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.103500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:39.103665 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.104225 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:39.104240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:39.105808 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:39.107925 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:39.107930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:39.107941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:39.108185 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:39.108523 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:39.108529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:39.108533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:39.108747 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:39.109129 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:39.109137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:39.109514 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 16:38:39.109743 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 16:38:39.110075 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260925 16:38:39.112221 6292 heartbeater.cc:347] Connected to a master server at 127.0.97.254:35333
I20260925 16:38:39.112464 6292 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:39.112994 6292 heartbeater.cc:511] Master 127.0.97.254:35333 requested a full tablet report, sending...
I20260925 16:38:39.114108 5967 ts_manager.cc:194] Registered new tserver with Master: c8bf899d5a194184985cbfdd3bb279fd (127.0.97.194:42225)
I20260925 16:38:39.114730 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:32943
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:39.132861 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:39.243228 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:39.243537 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:39.243616 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:39.247732 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:39.247831 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:39.247924 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:39.253266 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:35333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:39.254383 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:39.255403 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:39.265254 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:39.265280 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:39.265507 6297 server_base.cc:1034] running on GCE node
W20260925 16:38:39.265260 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:39.266057 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:39.266676 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:39.267834 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790354319267820 us; error 27 us; skew 500 ppm
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.270556 6297 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:39.271426 6297 webserver.cc:460] Webserver started at http://127.0.97.195:43581/ using document root <none> and password file <none>
I20260925 16:38:39.272014 6297 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:39.272075 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:39.272317 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:39.274369 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "d2165baaa09e4b0f819be695bae7d42e"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "63578359756e775b5ec71cf90f870eb9"
server_key_iv: "ccda5026c095b3035ef802b5e5ad1ae0"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.274935 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "d2165baaa09e4b0f819be695bae7d42e"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "63578359756e775b5ec71cf90f870eb9"
server_key_iv: "ccda5026c095b3035ef802b5e5ad1ae0"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.278635 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:39.281011 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.282233 6297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:39.282367 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "d2165baaa09e4b0f819be695bae7d42e"
format_stamp: "Formatted at 2026-09-25 16:38:39 on dist-test-slave-kxl8"
server_key: "63578359756e775b5ec71cf90f870eb9"
server_key_iv: "ccda5026c095b3035ef802b5e5ad1ae0"
server_key_version: "encryptionkey@0"
I20260925 16:38:39.282474 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:39.311688 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:39.314679 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:39.314888 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:39.315503 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:39.316357 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:39.316425 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.316496 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:39.316550 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:39.325574 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:36029
I20260925 16:38:39.325593 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:36029 every 8 connection(s)
I20260925 16:38:39.326495 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:39.329602 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6297
I20260925 16:38:39.329757 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:39.330045 391 external_mini_cluster.cc:1428] Setting key 497da9735f445d7174ed36d325ad2493
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:39.340021 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.328328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:54383 (local address 127.0.97.254:35333)
0925 16:38:39.328463 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.328467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.330383 (+ 1916us) server_negotiation.cc:407] Connection header received
0925 16:38:39.330896 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.330899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.330952 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.331060 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.332208 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.332802 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.333994 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.334213 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.334323 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.336667 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:39.336671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:39.336675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:39.336703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:39.338039 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.338550 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.338555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.338557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.338636 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.338967 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.338972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.338974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.339145 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:39.339253 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:39.339734 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 16:38:39.339857 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:38:39.340807 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.328714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:35333 (local address 127.0.97.195:54383)
0925 16:38:39.329170 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:39.329214 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:39.330667 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:39.331232 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:39.331239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:39.331594 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:39.332061 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:39.332073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.332946 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.332949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:39.333872 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:39.333883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.334365 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.334369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:39.334533 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.335067 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:39.335082 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:39.336514 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:39.338155 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:39.338161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:39.338176 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:39.338446 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:39.338729 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:39.338732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:39.338734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:39.338873 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:39.339249 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:39.339255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:39.339509 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 16:38:39.339695 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:38:39.340007 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260925 16:38:39.341919 6426 heartbeater.cc:347] Connected to a master server at 127.0.97.254:35333
I20260925 16:38:39.342191 6426 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:39.342748 6426 heartbeater.cc:511] Master 127.0.97.254:35333 requested a full tablet report, sending...
I20260925 16:38:39.343832 5967 ts_manager.cc:194] Registered new tserver with Master: d2165baaa09e4b0f819be695bae7d42e (127.0.97.195:36029)
I20260925 16:38:39.344399 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:54383
I20260925 16:38:39.355420 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/test-admin.keytab.
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.370261 391 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 16:38:39.373068 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:39.371559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.0.97.193:43493)
0925 16:38:39.371884 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.371902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.371929 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:38:39.371984 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.371989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.372176 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.372333 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.372855 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41934: BlockingRecv error: recv got EOF from 127.0.0.1:41934 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
W20260925 16:38:39.373157 391 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.374178 391 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.375164 391 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:39.384536 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.376000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.0.97.193:43493)
0925 16:38:39.376117 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.376121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.376225 (+ 104us) server_negotiation.cc:407] Connection header received
0925 16:38:39.376388 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.376390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.376427 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.376509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.377032 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.377871 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.378688 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.379070 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.379188 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.381367 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:39.381374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:39.381389 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:39.381411 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:39.382813 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.383239 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.383245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.383246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.383326 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.383601 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.383605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.383607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.383840 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:39.384055 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:39.384231 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 16:38:39.384398 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 16:38:39.388238 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:39.387424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.97.193:43493)
0925 16:38:39.387554 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.387557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.387571 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:39.387678 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.387680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.387713 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.387786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.388135 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41948: BlockingRecv error: recv got EOF from 127.0.0.1:41948 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 16:38:39.388288 391 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.389346 391 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.390376 391 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:39.398528 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.391122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.97.193:43493)
0925 16:38:39.391238 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.391242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.391343 (+ 101us) server_negotiation.cc:407] Connection header received
0925 16:38:39.391475 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.391478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.391513 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.391601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.392053 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.392557 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.393325 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.393583 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.393690 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.395699 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:39.395703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:39.395706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:39.395729 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:39.396915 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.397368 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.397371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.397373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.397448 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.397720 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.397723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.397724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.397891 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:39.398021 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:39.398265 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 16:38:39.398373 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:38:39.401258 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:39.400294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.97.193:43493)
0925 16:38:39.400407 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.400412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.400498 (+ 86us) server_negotiation.cc:407] Connection header received
0925 16:38:39.400634 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.400640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.400676 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.400766 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.401134 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41980: BlockingRecv error: recv got EOF from 127.0.0.1:41980 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 16:38:39.401443 391 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790354288967115-391-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.402418 391 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:39.403249 391 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:38:39.411132 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:39.404021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.97.193:43493)
0925 16:38:39.404140 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:39.404143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:39.404191 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:38:39.404273 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:39.404275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:39.404305 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:39.404368 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:39.404782 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.405325 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.406057 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:39.406258 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:39.406331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:39.408487 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:39.408489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:39.408491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:39.408510 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:39.409751 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.410119 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.410121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.410123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.410189 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:39.410414 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:39.410416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:39.410417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:39.410562 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:39.410652 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:39.410844 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 16:38:39.410954 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:38:39.412238 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6029
I20260925 16:38:39.417790 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6163
I20260925 16:38:39.423069 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6297
I20260925 16:38:39.430166 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 5935
2026-09-25T16:38:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2183 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[6446](info): set up 2 sockets
Sep 25 16:38:39 dist-test-slave-kxl8 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260925 16:38:40.511050 391 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:40 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab.
Entry for principal oryx/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab.
I20260925 16:38:40.630414 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:37533
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab
--principal=oryx/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:37533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:40.739568 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:40.739866 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:40.739953 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:40.743964 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:40.744048 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:40.744087 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:40.744149 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:40.744189 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:40.749434 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:37533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab
--principal=oryx/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:37533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:40.750435 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:40.751420 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:40.760083 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:40.760100 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:40.760097 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:40.760473 6459 server_base.cc:1034] running on GCE node
I20260925 16:38:40.760962 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:40.762101 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:40.763268 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790354320763249 us; error 30 us; skew 500 ppm
Sep 25 16:38:40 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:40.766060 6459 init.cc:377] Logged in from keytab as oryx/127.0.97.254@KRBTEST.COM (short username oryx)
I20260925 16:38:40.766882 6459 webserver.cc:460] Webserver started at http://127.0.97.254:45243/ using document root <none> and password file <none>
I20260925 16:38:40.767414 6459 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:40.767473 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:40.767724 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:40.769810 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "897240c44b18481e87b83a49e1cfd688"
format_stamp: "Formatted at 2026-09-25 16:38:40 on dist-test-slave-kxl8"
server_key: "00fc58b975841af3b489b2d9dca97ad7"
server_key_iv: "cca4f303772ec752e11d95075ded075d"
server_key_version: "encryptionkey@0"
I20260925 16:38:40.770354 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "897240c44b18481e87b83a49e1cfd688"
format_stamp: "Formatted at 2026-09-25 16:38:40 on dist-test-slave-kxl8"
server_key: "00fc58b975841af3b489b2d9dca97ad7"
server_key_iv: "cca4f303772ec752e11d95075ded075d"
server_key_version: "encryptionkey@0"
I20260925 16:38:40.774026 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:38:40.776549 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:40.777781 6459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:40.777928 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "897240c44b18481e87b83a49e1cfd688"
format_stamp: "Formatted at 2026-09-25 16:38:40 on dist-test-slave-kxl8"
server_key: "00fc58b975841af3b489b2d9dca97ad7"
server_key_iv: "cca4f303772ec752e11d95075ded075d"
server_key_version: "encryptionkey@0"
I20260925 16:38:40.778043 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:40.795087 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:40.798306 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:40.798513 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:40.805073 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:37533
I20260925 16:38:40.805414 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:37533 every 8 connection(s)
I20260925 16:38:40.806034 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:40.808773 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:40.813985 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Bootstrap starting.
I20260925 16:38:40.816171 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6459
I20260925 16:38:40.816329 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:40.816545 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:40.816607 391 external_mini_cluster.cc:1428] Setting key 2ad672935fae30d99ea398f3f68350fd
I20260925 16:38:40.817415 6527 log.cc:824] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:40.819247 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: No bootstrap required, opened a new log
I20260925 16:38:40.822625 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } }
I20260925 16:38:40.822868 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:40.822960 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897240c44b18481e87b83a49e1cfd688, State: Initialized, Role: FOLLOWER
I20260925 16:38:40.823431 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } }
I20260925 16:38:40.823576 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:40.823662 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:40.823763 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:40.824635 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } }
Sep 25 16:38:40 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:40.825023 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897240c44b18481e87b83a49e1cfd688; no voters:
I20260925 16:38:40.825403 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:40.825497 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:40.825707 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 LEADER]: Becoming Leader. State: Replica: 897240c44b18481e87b83a49e1cfd688, State: Running, Role: LEADER
I20260925 16:38:40.826061 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } }
I20260925 16:38:40.826905 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:40.827358 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897240c44b18481e87b83a49e1cfd688. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } }
I20260925 16:38:40.827478 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:40.827844 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:40.827791 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } }
I20260925 16:38:40.827921 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:40.831082 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:40.831782 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:40.817720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.97.254:37533)
0925 16:38:40.818153 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:40.818168 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:40.818203 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:38:40.818811 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:40.818827 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:40.819127 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:40.819460 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:40.820234 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:40.821186 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:40.822035 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:40.822499 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:40.822666 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:40.825492 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:40.825499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:40.825519 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:40.825554 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:40.827766 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:40.828260 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:40.828266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:40.828271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:40.828395 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:40.828719 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:40.828722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:40.828723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:40.829216 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:40.829426 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:40.830372 (+ 946us) server_negotiation.cc:299] Negotiation successful
0925 16:38:40.830651 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260925 16:38:40.837059 6536 catalog_manager.cc:1347] Generated new cluster ID: e78cbf005d91467baf68de4720ddd273
I20260925 16:38:40.837116 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:40.854948 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:40.855865 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:40.867354 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Generated new TSK 0
I20260925 16:38:40.868155 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
I20260925 16:38:40.900454 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:41.011188 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:41.011467 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:41.011510 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:41.015547 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:41.015604 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:41.015666 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:41.020934 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:41.021970 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:41.022919 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:41.032668 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.032837 6553 server_base.cc:1034] running on GCE node
W20260925 16:38:41.032763 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.032709 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.033344 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:41.033993 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:41.035171 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790354321035156 us; error 32 us; skew 500 ppm
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:41.037890 6553 init.cc:377] Logged in from keytab as oryx/127.0.97.193@KRBTEST.COM (short username oryx)
I20260925 16:38:41.038743 6553 webserver.cc:460] Webserver started at http://127.0.97.193:36619/ using document root <none> and password file <none>
I20260925 16:38:41.039311 6553 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.039400 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.039621 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:41.041693 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "60ab6503654145068332e88f3ea709ac"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "583f2c3a2293290fa1c476570847c2e6"
server_key_iv: "b6c0059e29c2413bdef0e3d2c6e10cd2"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.042227 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "60ab6503654145068332e88f3ea709ac"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "583f2c3a2293290fa1c476570847c2e6"
server_key_iv: "b6c0059e29c2413bdef0e3d2c6e10cd2"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.045794 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:41.048067 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.049244 6553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:41.049432 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "60ab6503654145068332e88f3ea709ac"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "583f2c3a2293290fa1c476570847c2e6"
server_key_iv: "b6c0059e29c2413bdef0e3d2c6e10cd2"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.049539 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.065853 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:41.068656 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:41.068868 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:41.069469 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:41.070240 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:41.070318 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.070399 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:41.070449 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.079419 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:34965
I20260925 16:38:41.079442 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:34965 every 8 connection(s)
I20260925 16:38:41.080266 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:41.086748 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6553
I20260925 16:38:41.086879 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:41.087149 391 external_mini_cluster.cc:1428] Setting key 7215061008b903258bee5c7d226de8cc
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:41.094841 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.082096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:37273 (local address 127.0.97.254:37533)
0925 16:38:41.082254 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.082258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.083105 (+ 847us) server_negotiation.cc:407] Connection header received
0925 16:38:41.084450 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.084453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.084511 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.084612 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.085825 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.086367 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.087732 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.088050 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.088140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.090727 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.090734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.090740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.090775 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.092542 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.093019 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.093025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.093028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.093126 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.093522 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.093528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.093532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.093804 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.093929 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.094510 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.094651 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:38:41.095633 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.082537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:37273)
0925 16:38:41.082965 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.083005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.084218 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.084788 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.084794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.085141 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.085681 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.085694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.086490 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.086494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.087611 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.087625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.088198 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.088202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.088425 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.089095 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.089111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.090570 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.092658 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.092662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.092673 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.092913 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.093211 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.093213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.093214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.093417 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.093954 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.093960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.094225 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.094411 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.094699 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:41.097116 6682 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:41.097440 6682 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:41.097968 6682 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:41.099663 6491 ts_manager.cc:194] Registered new tserver with Master: 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:41.101032 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:37273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
I20260925 16:38:41.116686 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:41.231455 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:41.231750 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:41.231841 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:41.235873 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:41.235962 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:41.236056 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:41.241142 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:41.242208 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:41.243305 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:41.252976 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.253054 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.253216 6687 server_base.cc:1034] running on GCE node
W20260925 16:38:41.253054 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.253671 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:41.254352 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:41.255548 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790354321255505 us; error 58 us; skew 500 ppm
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:41.262243 6687 init.cc:377] Logged in from keytab as oryx/127.0.97.194@KRBTEST.COM (short username oryx)
I20260925 16:38:41.263172 6687 webserver.cc:460] Webserver started at http://127.0.97.194:32909/ using document root <none> and password file <none>
I20260925 16:38:41.263754 6687 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.263847 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.264081 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:41.266266 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "ed1e284ddd77479ca69c1288e0c15f42"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "2a3e1089228d472b0278b8016e778892"
server_key_iv: "301494b3d5ab314468a5c351ace5e0a4"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.266824 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "ed1e284ddd77479ca69c1288e0c15f42"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "2a3e1089228d472b0278b8016e778892"
server_key_iv: "301494b3d5ab314468a5c351ace5e0a4"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.270459 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:41.272701 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.273978 6687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:41.274106 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "ed1e284ddd77479ca69c1288e0c15f42"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "2a3e1089228d472b0278b8016e778892"
server_key_iv: "301494b3d5ab314468a5c351ace5e0a4"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.274209 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.287794 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:41.290664 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:41.290876 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:41.291476 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:41.292416 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:41.292490 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.292543 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:41.292577 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.301285 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38797
I20260925 16:38:41.301358 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38797 every 8 connection(s)
I20260925 16:38:41.302191 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:41.302999 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6687
I20260925 16:38:41.303100 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:41.303426 391 external_mini_cluster.cc:1428] Setting key 00143aa308a76d012852922b445da2b8
WARNING: no policy specified for oryx/127.0.97.195@KRBTEST.COM; defaulting to no policy
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.195@KRBTEST.COM" created.
I20260925 16:38:41.318003 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.304107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:50719 (local address 127.0.97.254:37533)
0925 16:38:41.304254 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.304259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.305752 (+ 1493us) server_negotiation.cc:407] Connection header received
0925 16:38:41.306946 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.306952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.307011 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.307107 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.308853 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.309465 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.311328 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.311540 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.311618 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.314410 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.314413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.314415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.314437 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.315763 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.316243 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.316245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.316246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.316344 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.316762 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.316769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.316773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.316986 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.317068 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.317767 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.317886 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:41.319049 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.304601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:50719)
0925 16:38:41.305579 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.305637 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.306740 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.307251 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.307258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.307741 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.308574 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.308599 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.309599 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.309606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.311201 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.311214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.311703 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.311707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.311920 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.312539 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.312563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.314252 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.315900 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.315905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.315915 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.316158 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.316452 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.316454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.316456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.316583 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.317099 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.317109 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.317483 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.317699 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.318072 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":608,"threads_started":1}
I20260925 16:38:41.320346 6816 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:41.320629 6816 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:41.321229 6816 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:41.322436 6491 ts_manager.cc:194] Registered new tserver with Master: ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:41.322995 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:50719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
I20260925 16:38:41.332470 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:41.444001 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:41.444235 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:41.444307 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:41.448295 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:41.448345 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:41.448439 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:41.453975 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:41.454960 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:41.455958 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:41.465632 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.465601 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.465616 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.465677 6821 server_base.cc:1034] running on GCE node
I20260925 16:38:41.466241 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:41.466919 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:41.468109 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790354321468095 us; error 26 us; skew 500 ppm
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:41.470963 6821 init.cc:377] Logged in from keytab as oryx/127.0.97.195@KRBTEST.COM (short username oryx)
I20260925 16:38:41.471889 6821 webserver.cc:460] Webserver started at http://127.0.97.195:39583/ using document root <none> and password file <none>
I20260925 16:38:41.472455 6821 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.472544 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.472777 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:41.474910 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "8e2648a21186e26785ca6e2056addd33"
server_key_iv: "aa752ff1de0647e8e3e6556c1cddd390"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.475471 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "8e2648a21186e26785ca6e2056addd33"
server_key_iv: "aa752ff1de0647e8e3e6556c1cddd390"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.479315 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:38:41.481789 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.483122 6821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:41.483268 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "8e2648a21186e26785ca6e2056addd33"
server_key_iv: "aa752ff1de0647e8e3e6556c1cddd390"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.483386 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.496665 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:41.499603 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:41.499809 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:41.500384 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:41.501205 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:41.501274 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.501385 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:41.501437 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.510360 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:32799
I20260925 16:38:41.510377 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:32799 every 8 connection(s)
I20260925 16:38:41.511242 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:41.519439 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6821
I20260925 16:38:41.519565 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:41.519860 391 external_mini_cluster.cc:1428] Setting key a40c62883bacc84dafe0440a7c87f719
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:41.524631 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.512991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:38925 (local address 127.0.97.254:37533)
0925 16:38:41.513135 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.513139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.514029 (+ 890us) server_negotiation.cc:407] Connection header received
0925 16:38:41.515296 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.515299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.515345 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.515437 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.516549 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.517131 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.518237 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.518456 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.518548 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.521044 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.521048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.521051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.521083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.522645 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.523145 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.523149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.523151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.523235 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.523579 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.523582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.523584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.523768 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.523888 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.524360 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.524478 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:38:41.525492 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.513454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:38925)
0925 16:38:41.513879 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.513920 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.515080 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.515592 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.515598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.515943 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.516412 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.516424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.517239 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.517242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.518127 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.518136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.518588 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.518590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.518724 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.519326 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.519348 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.520886 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.522761 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.522767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.522782 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.523039 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.523340 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.523346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.523351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.523475 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.523888 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.523896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.524133 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.524322 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.524644 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260925 16:38:41.526686 6950 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:41.526952 6950 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:41.527547 6950 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:41.528625 6491 ts_manager.cc:194] Registered new tserver with Master: 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
I20260925 16:38:41.529203 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:38925
I20260925 16:38:41.534024 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.253.keytab.
Entry for principal oryx/127.0.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.253.keytab.
I20260925 16:38:41.560318 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40879
--webserver_interface=127.0.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.253.keytab
--principal=oryx/127.0.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:37533,127.0.97.253:40879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:41.669780 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:41.670032 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:41.670071 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:41.673954 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:41.674005 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:41.674024 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:41.674041 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:41.674055 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:41.679227 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:37533,127.0.97.253:40879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.253.keytab
--principal=oryx/127.0.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
W20260925 16:38:41.679428 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.254:37533,127.0.97.253:40879'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 16:38:41.680228 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:41.681174 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:41.689906 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.690007 6955 server_base.cc:1034] running on GCE node
W20260925 16:38:41.689906 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.689937 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.690796 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:41.691972 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:41.693147 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790354321693113 us; error 52 us; skew 500 ppm
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:41.695990 6955 init.cc:377] Logged in from keytab as oryx/127.0.97.253@KRBTEST.COM (short username oryx)
I20260925 16:38:41.696946 6955 webserver.cc:460] Webserver started at http://127.0.97.253:33525/ using document root <none> and password file <none>
I20260925 16:38:41.697604 6955 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.697669 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.697881 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:41.699977 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/instance:
uuid: "3818a2fff337466fbdc4d305e38beb60"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "4397d9551285557695cc322b50187788"
server_key_iv: "5b90c50db07d2eb23e76cd9c759115c9"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.700554 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal/instance:
uuid: "3818a2fff337466fbdc4d305e38beb60"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "4397d9551285557695cc322b50187788"
server_key_iv: "5b90c50db07d2eb23e76cd9c759115c9"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.704438 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:38:41.706966 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.708263 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:41.708420 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "3818a2fff337466fbdc4d305e38beb60"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "4397d9551285557695cc322b50187788"
server_key_iv: "5b90c50db07d2eb23e76cd9c759115c9"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.708544 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.723744 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:41.727104 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:41.727310 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:41.734866 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.253:40879
I20260925 16:38:41.735179 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.253:40879 every 8 connection(s)
I20260925 16:38:41.735857 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
I20260925 16:38:41.738723 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:41.742545 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:38:41.746196 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 6955
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.253@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:41.757647 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.744419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.0.97.254:37533)
0925 16:38:41.744588 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.744592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.745448 (+ 856us) server_negotiation.cc:407] Connection header received
0925 16:38:41.746582 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.746585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.746631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.746707 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.748220 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.748874 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.750118 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.750326 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.750405 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.753510 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.753516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.753519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.753552 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.755302 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.755865 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.755868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.755870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.755964 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.756355 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.756357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.756359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.756687 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.756796 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.757403 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.757525 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:38:41.758723 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.747098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.0.97.253:40879)
0925 16:38:41.747421 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.747431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.747451 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:38:41.747538 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.747542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.747698 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.747824 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.748635 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.749737 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.750719 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.751156 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.751270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.754077 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.754084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.754099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.754127 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.755927 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.756439 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.756445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.756446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.756522 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.756771 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.756774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.756776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.757062 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.757212 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.758027 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.758161 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":216,"spinlock_wait_cycles":13312,"thread_start_us":115,"threads_started":1}
I20260925 16:38:41.758723 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.744779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46588)
0925 16:38:41.745231 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.745272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.746341 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.746913 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.746923 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.747421 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.748061 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.748082 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.749001 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.749004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.749997 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.750008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.750466 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.750470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.750667 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.751387 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.751413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.753313 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.755416 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.755423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.755432 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.755746 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.756062 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.756065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.756067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.756254 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.756810 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.756817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.757075 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.757343 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.757689 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":48,"spinlock_wait_cycles":24704,"thread_start_us":110,"threads_started":1}
I20260925 16:38:41.760156 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.253@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:41.767796 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.760659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49076)
0925 16:38:41.760816 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.760832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.760895 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.761133 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.761137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.761371 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.761459 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.761464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.762248 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.762251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.762750 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.762756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.763206 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.763209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.763307 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.763721 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.763731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.764863 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.766381 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.766384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.766386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.766580 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.766863 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.766869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.766872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.766950 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.767297 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.767299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.767377 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.767484 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.767616 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 16:38:41.767798 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.760754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.0.97.253:40879)
0925 16:38:41.760869 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.760873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.760887 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:41.760985 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.760988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.761036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.761127 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.761572 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.762119 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.762877 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.763106 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.763211 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.764982 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.764986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.764988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.765016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.766258 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.766687 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.766691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.766694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.766767 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.767034 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.767041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.767042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.767202 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.767307 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.767534 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.767670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":45}
I20260925 16:38:41.771198 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Bootstrap starting.
I20260925 16:38:41.773739 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:41.774470 7023 log.cc:824] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:41.776357 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: No bootstrap required, opened a new log
I20260925 16:38:41.779600 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } }
I20260925 16:38:41.779829 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:41.779865 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3818a2fff337466fbdc4d305e38beb60, State: Initialized, Role: FOLLOWER
I20260925 16:38:41.780365 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } }
I20260925 16:38:41.780893 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } } }
I20260925 16:38:41.781009 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:41.781679 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:38:41.786885 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:38:41.786986 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:38:41.788410 6955 master_runner.cc:428] Detected that this master 3818a2fff337466fbdc4d305e38beb60 is joining an existing cluster
I20260925 16:38:41.788475 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.253:40879
I20260925 16:38:41.799371 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.790601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.0.97.254:37533)
0925 16:38:41.790756 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.790761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.790947 (+ 186us) server_negotiation.cc:407] Connection header received
0925 16:38:41.791152 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.791156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.791250 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.791316 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.792164 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.792938 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.793861 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.794096 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.794211 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.795438 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.795443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.795446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.795478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.797304 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.797889 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.797893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.797895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.797994 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798356 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.798360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.798362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.798605 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.798685 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.799062 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.799188 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:38:41.799566 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.790518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46602)
0925 16:38:41.790843 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.790861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.791026 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.791495 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.791498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.791798 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.792000 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.792009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.793054 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.793063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.793735 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.793743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.794219 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.794222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.794309 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.794863 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.794875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.795264 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.797416 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.797420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.797422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.797772 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.798107 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.798113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.798225 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.798769 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.798772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.798872 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.798994 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.799171 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":21,"thread_start_us":116,"threads_started":1}
I20260925 16:38:41.799702 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.791017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.253:40879)
0925 16:38:41.791111 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.791115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.791389 (+ 274us) server_negotiation.cc:407] Connection header received
0925 16:38:41.791500 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.791506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.791541 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.791647 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.792108 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.792884 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.794250 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.794628 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.794704 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.796417 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.796423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.796429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.796457 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.798182 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798564 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.798570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.798572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.798643 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798909 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.798912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.798913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.799118 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.799207 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.799471 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.799577 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 16:38:41.800171 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.791076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49078)
0925 16:38:41.791290 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.791302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.791425 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.791647 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.791650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.791891 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.792001 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.792007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.793007 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.793011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.794096 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.794107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.794845 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.794849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.794958 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.795804 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.795821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.796267 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.798272 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.798282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.798470 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.798731 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.798736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.798738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.798818 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.799239 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.799242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.799314 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.799861 (+ 547us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.800006 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.254@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:41.815362 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.804384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.0.97.253:40879)
0925 16:38:41.804497 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.804503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.805337 (+ 834us) server_negotiation.cc:407] Connection header received
0925 16:38:41.805399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.805402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.805452 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.805530 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.806337 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.807039 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.808407 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.808763 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.808860 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.811621 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.811624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.811627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.811649 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.813450 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.813970 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.813974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.813976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.814051 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.814358 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.814361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.814363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.814570 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.814639 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.815079 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.815183 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:41.815879 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.804358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49086)
0925 16:38:41.804701 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.804742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.804917 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.805588 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.805595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.806063 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.806195 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.806214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.807155 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.807158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.808256 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.808267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.808879 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.808884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.809017 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.809602 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.809618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.811455 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.813570 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.813578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.813584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.813860 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.814160 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.814162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.814164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.814245 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.814717 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.814724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.814869 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.815535 (+ 666us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.815726 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260925 16:38:41.817364 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.253:40879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3818a2fff337466fbdc4d305e38beb60"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.253"
port: 40879
}
attrs {
promote: true
}
}
dest_uuid: "897240c44b18481e87b83a49e1cfd688"
cas_config_opid_index: -1
I20260925 16:38:41.818041 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } }
I20260925 16:38:41.820362 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3818a2fff337466fbdc4d305e38beb60 (127.0.97.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } }
I20260925 16:38:41.820998 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:41.822193 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 16:38:41.822625 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.253:40879
I20260925 16:38:41.823117 6955 master.cc:537] Master@127.0.97.253:40879 shutting down...
I20260925 16:38:41.823129 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.97.252@KRBTEST.COM; defaulting to no policy
I20260925 16:38:41.825496 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } }
I20260925 16:38:41.825686 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.252@KRBTEST.COM" created.
W20260925 16:38:41.826102 6478 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 16:38:41.827785 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 -> Peer 3818a2fff337466fbdc4d305e38beb60 (127.0.97.253:40879): Couldn't send request to peer 3818a2fff337466fbdc4d305e38beb60. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:38:41.829329 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:41.829627 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:41.829736 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: stopping tablet replica
I20260925 16:38:41.835042 6955 master.cc:552] Master@127.0.97.253:40879 shutdown complete.
I20260925 16:38:41.835099 6955 master_runner.cc:305] Clearing existing system tablet
I20260925 16:38:41.835381 6955 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.835443 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.836844 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 16:38:41.837774 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.838022 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.838158 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:41.838200 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "3818a2fff337466fbdc4d305e38beb60"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "4397d9551285557695cc322b50187788"
server_key_iv: "5b90c50db07d2eb23e76cd9c759115c9"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.839986 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:41.842811 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:38:41.842886 6955 log.cc:1196] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 16:38:41.843134 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Deleting consensus metadata
I20260925 16:38:41.843320 6955 master_runner.cc:315] Copying system tablet from 127.0.97.254:37533
I20260925 16:38:41.844535 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.254:37533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.252.keytab.
Entry for principal oryx/127.0.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.252.keytab.
I20260925 16:38:41.845782 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:35927
--webserver_interface=127.0.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.252.keytab
--principal=oryx/127.0.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:38:41.854455 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.845208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.0.97.254:37533)
0925 16:38:41.845450 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:41.845456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:41.845712 (+ 256us) server_negotiation.cc:407] Connection header received
0925 16:38:41.846174 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:41.846178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:41.846245 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:41.846304 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:41.846977 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.847899 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.849213 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.849602 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.849725 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.850890 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:41.850893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:41.850896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:41.850922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:41.852559 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.853040 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.853044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.853045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.853130 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:41.853518 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:41.853522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:41.853524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:41.853826 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:41.853919 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:41.854177 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 16:38:41.854284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 16:38:41.854467 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:41.845116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46614)
0925 16:38:41.845615 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:41.845632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:41.845790 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:41.846457 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:41.846460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:41.846692 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:41.846829 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.846836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.848053 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.848057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.849080 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:41.849091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:41.849731 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:41.849735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:41.849840 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:41.850399 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:41.850419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:41.850736 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:41.852680 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.852685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.852690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.852932 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.853254 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:41.853258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:41.853260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:41.853422 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:41.853909 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:41.853912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:41.854017 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 16:38:41.854127 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 16:38:41.854277 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":83,"threads_started":1}
I20260925 16:38:41.854980 6511 tablet_copy_service.cc:140] P 897240c44b18481e87b83a49e1cfd688: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3818a2fff337466fbdc4d305e38beb60 ({username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:46614)
I20260925 16:38:41.855160 6511 tablet_copy_service.cc:161] P 897240c44b18481e87b83a49e1cfd688: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3818a2fff337466fbdc4d305e38beb60 at {username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:46614: session id = 3818a2fff337466fbdc4d305e38beb60-00000000000000000000000000000000
I20260925 16:38:41.856534 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:38:41.858071 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:38:41.859930 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: tablet copy: Starting download of 0 data blocks...
I20260925 16:38:41.860132 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: tablet copy: Starting download of 1 WAL segments...
I20260925 16:38:41.861775 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:38:41.863405 6511 tablet_copy_service.cc:342] P 897240c44b18481e87b83a49e1cfd688: Request end of tablet copy session 3818a2fff337466fbdc4d305e38beb60-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:46614
I20260925 16:38:41.863517 6511 tablet_copy_service.cc:433] P 897240c44b18481e87b83a49e1cfd688: ending tablet copy session 3818a2fff337466fbdc4d305e38beb60-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3818a2fff337466fbdc4d305e38beb60
I20260925 16:38:41.870325 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:41.871922 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:41.872025 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.872320 6955 server_base.cc:1034] running on GCE node
W20260925 16:38:41.872454 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:41.872699 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:41.873008 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:41.874152 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790354321874148 us; error 23 us; skew 500 ppm
Sep 25 16:38:41 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:41.875595 6955 init.cc:377] Logged in from keytab as oryx/127.0.97.253@KRBTEST.COM (short username oryx)
I20260925 16:38:41.876075 6955 webserver.cc:460] Webserver started at http://127.0.97.253:46283/ using document root <none> and password file <none>
I20260925 16:38:41.876257 6955 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:41.876317 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:41.877506 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:41.878458 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:41.878696 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:41.878794 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "3818a2fff337466fbdc4d305e38beb60"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "4397d9551285557695cc322b50187788"
server_key_iv: "5b90c50db07d2eb23e76cd9c759115c9"
server_key_version: "encryptionkey@0"
I20260925 16:38:41.878901 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:41.888537 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:41.890662 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:41.895550 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.253:40879
I20260925 16:38:41.895731 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.253:40879 every 8 connection(s)
I20260925 16:38:41.896142 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
I20260925 16:38:41.897655 7128 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:38:41.898448 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Bootstrap starting.
I20260925 16:38:41.907271 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:38:41.907599 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Bootstrap complete.
I20260925 16:38:41.908208 7128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } }
I20260925 16:38:41.908376 7128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3818a2fff337466fbdc4d305e38beb60, State: Initialized, Role: LEARNER
I20260925 16:38:41.908493 7128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } }
I20260925 16:38:41.908838 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } }
I20260925 16:38:41.909000 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:38:41.908999 7128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:41.913044 7142 catalog_manager.cc:1259] Loaded cluster ID: e78cbf005d91467baf68de4720ddd273
I20260925 16:38:41.913125 7142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: loading cluster ID for follower catalog manager: success
I20260925 16:38:41.914666 7142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: acquiring CA information for follower catalog manager: success
I20260925 16:38:41.915735 7142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 16:38:42.017241 7064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:42.017561 7064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:42.017649 7064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:42.021728 7064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:42.021809 7064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:42.021844 7064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:42.021903 7064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:42.021956 7064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:42.027295 7064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.252.keytab
--principal=oryx/127.0.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:42.028506 7064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:42.029659 7064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:42.038136 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.038136 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.038132 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.038174 7064 server_base.cc:1034] running on GCE node
I20260925 16:38:42.038918 7064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:42.040066 7064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:42.041266 7064 hybrid_clock.cc:648] HybridClock initialized: now 1790354322041255 us; error 26 us; skew 500 ppm
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:42.044255 7064 init.cc:377] Logged in from keytab as oryx/127.0.97.252@KRBTEST.COM (short username oryx)
I20260925 16:38:42.045143 7064 webserver.cc:460] Webserver started at http://127.0.97.252:42965/ using document root <none> and password file <none>
I20260925 16:38:42.045780 7064 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.045874 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.046109 7064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:42.048120 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/instance:
uuid: "b7ae709c2e094ba7a4a4160da30206e7"
format_stamp: "Formatted at 2026-09-25 16:38:42 on dist-test-slave-kxl8"
server_key: "b8d8a27d29818a8e10eb476f09a8a5e4"
server_key_iv: "b9cd819101ae4d45718f8a98004eb437"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.048697 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal/instance:
uuid: "b7ae709c2e094ba7a4a4160da30206e7"
format_stamp: "Formatted at 2026-09-25 16:38:42 on dist-test-slave-kxl8"
server_key: "b8d8a27d29818a8e10eb476f09a8a5e4"
server_key_iv: "b9cd819101ae4d45718f8a98004eb437"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.052600 7064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 16:38:42.055493 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.056711 7064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 16:38:42.056852 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "b7ae709c2e094ba7a4a4160da30206e7"
format_stamp: "Formatted at 2026-09-25 16:38:42 on dist-test-slave-kxl8"
server_key: "b8d8a27d29818a8e10eb476f09a8a5e4"
server_key_iv: "b9cd819101ae4d45718f8a98004eb437"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.056962 7064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.068046 7064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:42.071307 7064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:42.071503 7064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:42.079272 7064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.252:35927
I20260925 16:38:42.079560 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.252:35927 every 8 connection(s)
I20260925 16:38:42.080457 7064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
I20260925 16:38:42.083398 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:42.084959 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7064
I20260925 16:38:42.088311 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.252@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:42.099768 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.090765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.254:37533)
0925 16:38:42.090895 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.090899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.091175 (+ 276us) server_negotiation.cc:407] Connection header received
0925 16:38:42.091302 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.091304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.091348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.091408 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.092240 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.092801 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.093778 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.093997 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.094106 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.096494 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.096501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.096507 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.096540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.098058 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.098480 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.098485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.098487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.098570 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.098867 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.098872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.098874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.099044 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.099146 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.099497 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.099604 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:38:42.100113 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.086378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.97.252:35927)
0925 16:38:42.086817 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.086833 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.086867 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:38:42.087600 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.087618 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.087959 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.088242 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.089604 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.090856 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.092143 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.092627 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.092826 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.095438 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.095443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.095452 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.095471 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.097435 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.097832 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.097839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.097846 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.097975 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.098243 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.098248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.098250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.098619 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.098845 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.099100 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.099346 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260925 16:38:42.100111 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.090585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46624)
0925 16:38:42.091021 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.091073 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.091198 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.091484 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.091490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.091917 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.092099 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.092107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.092933 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.092936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.093656 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.093664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.094139 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.094143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.094246 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.094744 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.094759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.096330 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.098145 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.098154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.098159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.098369 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.098667 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.098670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.098672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.098749 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.099145 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.099152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.099275 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.099437 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.099606 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260925 16:38:42.101612 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:38:42.103539 6682 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.252@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:42.109831 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.102073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49092)
0925 16:38:42.102191 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.102207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.102300 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.102747 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.102749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.102894 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.103290 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.103295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.104359 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.104363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.104990 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.104998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.105461 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.105464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.105532 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.105974 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.105985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.107114 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.108562 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.108565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.108566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.108754 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.108976 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.108978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.108980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.109051 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.109487 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.109489 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.109558 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.109639 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.109725 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 16:38:42.109934 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.102195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.97.253:40879)
0925 16:38:42.102550 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.102561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.102572 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:38:42.102606 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.102608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.102644 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.102716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.103427 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.104249 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.105091 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.105372 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.105452 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.107231 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.107234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.107236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.107257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.108473 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.108834 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.108837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.108838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.108904 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.109122 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.109124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.109125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.109408 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.109484 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.109692 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.109779 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
I20260925 16:38:42.110625 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.252@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
I20260925 16:38:42.118113 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.111029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42102)
0925 16:38:42.111295 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.111309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.111381 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.111616 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.111619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.111767 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.111821 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.111825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.112531 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.112534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.113015 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.113021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.113517 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.113520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.113592 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.114021 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.114033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.115078 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.116668 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.116672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.116674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.116858 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.117109 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.117114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.117116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.117191 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.117559 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.117561 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.117637 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.117747 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.117907 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260925 16:38:42.118124 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.111185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.0.97.252:35927)
0925 16:38:42.111284 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.111288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.111350 (+ 62us) server_negotiation.cc:407] Connection header received
0925 16:38:42.111473 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.111476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.111523 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.111607 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.111932 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.112413 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.113125 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.113418 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.113522 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.115201 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.115204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.115207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.115227 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.116547 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.116948 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.116955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.116957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.117025 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.117302 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.117305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.117307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.117463 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.117559 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.117800 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.117974 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":37}
I20260925 16:38:42.120927 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Bootstrap starting.
I20260925 16:38:42.123167 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:42.123814 7210 log.cc:824] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:42.125582 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: No bootstrap required, opened a new log
I20260925 16:38:42.128717 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } }
I20260925 16:38:42.128907 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:42.128997 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ae709c2e094ba7a4a4160da30206e7, State: Initialized, Role: FOLLOWER
I20260925 16:38:42.129436 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } }
I20260925 16:38:42.129901 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } } }
I20260925 16:38:42.130015 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:42.130563 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:38:42.135280 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:38:42.135366 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:38:42.138422 7064 master_runner.cc:428] Detected that this master b7ae709c2e094ba7a4a4160da30206e7 is joining an existing cluster
I20260925 16:38:42.138485 7064 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.252:35927
I20260925 16:38:42.149422 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.141011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.97.253:40879)
0925 16:38:42.141134 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.141138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.141373 (+ 235us) server_negotiation.cc:407] Connection header received
0925 16:38:42.141535 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.141538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.141586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.141633 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.142268 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.142787 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.143765 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.144097 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.144186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.145737 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.145741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.145744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.145782 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.147608 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.148113 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.148117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.148119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.148206 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.148497 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.148501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.148503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.148724 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.148837 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.149114 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.149237 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:42.149446 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.140766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49106)
0925 16:38:42.141259 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.141306 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.141435 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.141766 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.141768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.141991 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.142127 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.142137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.142899 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.142902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.143632 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.143643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.144362 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.144366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.144459 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.145056 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.145081 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.145579 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.147744 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.147747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.147751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.148004 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.148310 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.148313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.148315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.148403 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.148825 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.148828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.148925 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.149043 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.149188 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":99,"threads_started":1}
I20260925 16:38:42.150266 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.140729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.0.97.254:37533)
0925 16:38:42.140919 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.140924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.141145 (+ 221us) server_negotiation.cc:407] Connection header received
0925 16:38:42.141373 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.141376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.141431 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.141488 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.142165 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.143024 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.144250 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.144582 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.145009 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.146559 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.146562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.146566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.146593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.148497 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.148906 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.148910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.148911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.148996 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.149318 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.149321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.149323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.149534 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.149607 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.149979 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.150102 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 16:38:42.150722 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.140646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46634)
0925 16:38:42.141059 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.141076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.141217 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.141640 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.141643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.141889 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.142030 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.142037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.143164 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.143167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.144114 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.144123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.145170 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.145173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.145264 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.145728 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.145741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.146414 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.148579 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.148582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.148584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.148818 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.149099 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.149102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.149104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.149185 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.149685 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.149689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.149799 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.150418 (+ 619us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.150556 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260925 16:38:42.153939 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6553
I20260925 16:38:42.161621 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:34965
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:37533
--webserver_port=36619
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.254@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
I20260925 16:38:42.169940 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.141146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.0.97.252:35927)
0925 16:38:42.141243 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.141247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.149666 (+ 8419us) server_negotiation.cc:407] Connection header received
0925 16:38:42.149928 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.149930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.149965 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.150010 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.150582 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.151230 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.152112 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.152448 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.152565 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.166412 (+ 13847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.166416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.166419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.166439 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.168264 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.168710 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.168713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.168715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.168782 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.169033 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.169036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.169038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.169249 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.169346 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.169661 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.169768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:38:42.170611 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.142068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42110)
0925 16:38:42.149605 (+ 7537us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.149621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.149835 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.150124 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.150127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.150297 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.150473 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.150480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.151345 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.151348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.151993 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.152001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.152557 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.152561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.152705 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.165795 (+ 13090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.165810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.166274 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.168373 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.168376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.168385 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.168602 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.168880 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.168883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.168885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.168954 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.169402 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.169406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.169510 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.170325 (+ 815us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.170456 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7473,"spinlock_wait_cycles":2432,"thread_start_us":91,"threads_started":1}
I20260925 16:38:42.171576 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.155392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42120)
0925 16:38:42.155490 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.155507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.155581 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.160856 (+ 5275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.160860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.161079 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.161178 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.161185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.162241 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.162245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.163104 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.163114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.163748 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.163754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.163861 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.164381 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.164396 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.166493 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.169651 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.169656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.169660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.169941 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.170334 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.170338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.170343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.170446 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.170928 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.170935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.171048 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.171157 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.171340 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 16:38:42.172191 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.155449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.0.97.252:35927)
0925 16:38:42.160472 (+ 5023us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.160483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.160501 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:38:42.160579 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.160585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.160651 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.160811 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.161348 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.162122 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.163229 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.163619 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.163738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.166956 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.166963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.166967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.167001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.169512 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.170081 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.170087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.170091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.170178 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.170573 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.170576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.170577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.170803 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.171748 (+ 945us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.171829 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.171959 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4930,"thread_start_us":101,"threads_started":1}
I20260925 16:38:42.173092 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.252:35927: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.252"
port: 35927
}
attrs {
promote: true
}
}
dest_uuid: "897240c44b18481e87b83a49e1cfd688"
cas_config_opid_index: 5
I20260925 16:38:42.173866 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } }
I20260925 16:38:42.174947 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.175450 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:42.176445 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.252:35927
I20260925 16:38:42.176609 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.176709 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.176832 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 16:38:42.176888 7064 master.cc:537] Master@127.0.97.252:35927 shutting down...
I20260925 16:38:42.178495 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:42.179970 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.175158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49120)
0925 16:38:42.175458 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.175471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.175552 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.175895 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.175901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.175919 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.176000 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.176005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.177023 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.177026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.178369 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.178380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.179566 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.179570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.179665 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.179712 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.179784 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.179850 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260925 16:38:42.180294 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.175314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.0.97.253:40879)
0925 16:38:42.175440 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.175444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.175512 (+ 68us) server_negotiation.cc:407] Connection header received
0925 16:38:42.175631 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.175634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.175692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.175779 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.176114 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.176868 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.178545 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.179228 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.179332 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.179483 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.180103 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.180176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:38:42.180717 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 LEARNER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260925 16:38:42.180779 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 -> Peer b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252:35927): Couldn't send request to peer b7ae709c2e094ba7a4a4160da30206e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:38:42.181155 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 16:38:42.182657 7064 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:42.182925 7064 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:42.183038 7064 tablet_replica.cc:330] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: stopping tablet replica
I20260925 16:38:42.183127 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.185540 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897240c44b18481e87b83a49e1cfd688. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.185667 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:38:42.185570 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: NON_VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: true } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.185810 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:38:42.186455 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: attempting to promote NON_VOTER 3818a2fff337466fbdc4d305e38beb60 to VOTER
I20260925 16:38:42.186874 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } }
I20260925 16:38:42.188539 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.188195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42132)
0925 16:38:42.188393 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.188463 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 16:38:42.188588 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 LEARNER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 16:38:42.189242 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 16:38:42.189908 7064 master.cc:552] Master@127.0.97.252:35927 shutdown complete.
I20260925 16:38:42.189954 7064 master_runner.cc:305] Clearing existing system tablet
I20260925 16:38:42.190178 7064 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.190217 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.190982 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3818a2fff337466fbdc4d305e38beb60 (127.0.97.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.192348 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.192442 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.192550 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.192612 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.193125 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.189069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42142)
0925 16:38:42.193003 (+ 3934us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.193056 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3864}
W20260925 16:38:42.193321 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 -> Peer b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252:35927): Couldn't send request to peer b7ae709c2e094ba7a4a4160da30206e7. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.252:35927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:38:42.193218 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3818a2fff337466fbdc4d305e38beb60 (127.0.97.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.194897 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.195006 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:42.201651 7064 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.002s
I20260925 16:38:42.202612 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.202852 7064 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.202977 7064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:42.203009 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "b7ae709c2e094ba7a4a4160da30206e7"
format_stamp: "Formatted at 2026-09-25 16:38:42 on dist-test-slave-kxl8"
server_key: "b8d8a27d29818a8e10eb476f09a8a5e4"
server_key_iv: "b9cd819101ae4d45718f8a98004eb437"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.204970 7064 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:42.207455 7064 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:38:42.207546 7064 log.cc:1196] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 16:38:42.207789 7064 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Deleting consensus metadata
I20260925 16:38:42.207955 7064 master_runner.cc:315] Copying system tablet from 127.0.97.254:37533
I20260925 16:38:42.209110 7064 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.254:37533
I20260925 16:38:42.218417 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.209760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.0.97.254:37533)
0925 16:38:42.209912 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.209917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.210376 (+ 459us) server_negotiation.cc:407] Connection header received
0925 16:38:42.210514 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.210518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.210575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.210679 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.211168 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.211980 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.213176 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.213547 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.213647 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.214942 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.214947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.214949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.214975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.216646 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.217117 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.217121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.217122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.217216 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.217530 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.217533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.217535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.217751 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.217850 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.218139 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.218259 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:42.218420 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.209894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:46648)
0925 16:38:42.210261 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.210277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.210407 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.210710 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.210713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.210932 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.211045 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.211051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.212096 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.212100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.213045 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.213056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.213674 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.213678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.213779 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.214361 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.214376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.214796 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.216758 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.216762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.216764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.217014 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.217340 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.217343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.217345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.217438 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.217851 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.217854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.217951 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.218059 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.218200 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20260925 16:38:42.219265 6511 tablet_copy_service.cc:140] P 897240c44b18481e87b83a49e1cfd688: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b7ae709c2e094ba7a4a4160da30206e7 ({username='oryx', principal='oryx/127.0.97.252@KRBTEST.COM'} at 127.0.0.1:46648)
I20260925 16:38:42.219376 6511 tablet_copy_service.cc:161] P 897240c44b18481e87b83a49e1cfd688: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b7ae709c2e094ba7a4a4160da30206e7 at {username='oryx', principal='oryx/127.0.97.252@KRBTEST.COM'} at 127.0.0.1:46648: session id = b7ae709c2e094ba7a4a4160da30206e7-00000000000000000000000000000000
I20260925 16:38:42.220422 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:38:42.221652 7064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:38:42.224190 7064 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: tablet copy: Starting download of 0 data blocks...
I20260925 16:38:42.224421 7064 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: tablet copy: Starting download of 1 WAL segments...
I20260925 16:38:42.227293 7064 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:38:42.229341 6511 tablet_copy_service.cc:342] P 897240c44b18481e87b83a49e1cfd688: Request end of tablet copy session b7ae709c2e094ba7a4a4160da30206e7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.252@KRBTEST.COM'} at 127.0.0.1:46648
I20260925 16:38:42.229423 6511 tablet_copy_service.cc:433] P 897240c44b18481e87b83a49e1cfd688: ending tablet copy session b7ae709c2e094ba7a4a4160da30206e7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b7ae709c2e094ba7a4a4160da30206e7
I20260925 16:38:42.236730 7064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:42.238749 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.238992 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.239194 7064 server_base.cc:1034] running on GCE node
W20260925 16:38:42.239555 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.239771 7064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:42.240084 7064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:42.241221 7064 hybrid_clock.cc:648] HybridClock initialized: now 1790354322241214 us; error 35 us; skew 500 ppm
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:42.242848 7064 init.cc:377] Logged in from keytab as oryx/127.0.97.252@KRBTEST.COM (short username oryx)
I20260925 16:38:42.243338 7064 webserver.cc:460] Webserver started at http://127.0.97.252:37015/ using document root <none> and password file <none>
I20260925 16:38:42.243526 7064 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.243566 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.244848 7064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:42.245965 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.246235 7064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:42.246313 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "b7ae709c2e094ba7a4a4160da30206e7"
format_stamp: "Formatted at 2026-09-25 16:38:42 on dist-test-slave-kxl8"
server_key: "b8d8a27d29818a8e10eb476f09a8a5e4"
server_key_iv: "b9cd819101ae4d45718f8a98004eb437"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.246408 7064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.265086 7064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:42.267553 7064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:42.272837 7064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.252:35927
I20260925 16:38:42.272888 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.252:35927 every 8 connection(s)
I20260925 16:38:42.273794 7064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
I20260925 16:38:42.275880 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:38:42.277098 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Bootstrap starting.
I20260925 16:38:42.289835 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:38:42.290230 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: Bootstrap complete.
I20260925 16:38:42.290854 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } }
I20260925 16:38:42.291015 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7ae709c2e094ba7a4a4160da30206e7, State: Initialized, Role: LEARNER
I20260925 16:38:42.291144 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } }
I20260925 16:38:42.291544 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.291684 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:42.291736 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:38:42.295969 7325 catalog_manager.cc:1259] Loaded cluster ID: e78cbf005d91467baf68de4720ddd273
I20260925 16:38:42.296031 7325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: loading cluster ID for follower catalog manager: success
I20260925 16:38:42.298514 7325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: acquiring CA information for follower catalog manager: success
I20260925 16:38:42.299890 7325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 16:38:42.312263 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:42.312497 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:42.312543 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:42.316511 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:42.316563 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:42.316653 7237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:42.321956 7237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:34965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=36619
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:42.323041 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:42.324064 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:38:42.325718 6816 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
W20260925 16:38:42.334372 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.334486 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.334622 7237 server_base.cc:1034] running on GCE node
W20260925 16:38:42.334542 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.335059 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:42.335665 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:42.336850 7237 hybrid_clock.cc:648] HybridClock initialized: now 1790354322336830 us; error 30 us; skew 500 ppm
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:42.339592 7237 init.cc:377] Logged in from keytab as oryx/127.0.97.193@KRBTEST.COM (short username oryx)
I20260925 16:38:42.340472 7237 webserver.cc:460] Webserver started at http://127.0.97.193:36619/ using document root <none> and password file <none>
I20260925 16:38:42.341001 7237 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.341070 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.344868 7237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:42.347090 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.348484 7237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:42.348575 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "60ab6503654145068332e88f3ea709ac"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "583f2c3a2293290fa1c476570847c2e6"
server_key_iv: "b6c0059e29c2413bdef0e3d2c6e10cd2"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.349100 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.362200 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:42.364838 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:42.364981 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:42.365571 7237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:42.366396 7237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:42.366441 7237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.366479 7237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:42.366494 7237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.375835 7237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:34965
I20260925 16:38:42.375862 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:34965 every 8 connection(s)
I20260925 16:38:42.376767 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:42.378538 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7237
I20260925 16:38:42.378705 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6687
I20260925 16:38:42.385962 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:38797
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:37533
--webserver_port=32909
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:42.396077 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.381018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:49623 (local address 127.0.97.253:40879)
0925 16:38:42.381188 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.381194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.381834 (+ 640us) server_negotiation.cc:407] Connection header received
0925 16:38:42.382494 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.382501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.382567 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.382624 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.384570 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385418 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387402 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.387790 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387891 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.392370 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.392376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.392381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.392412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.394274 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394745 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.394749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.394751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.394833 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.395114 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.395119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.395124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.395317 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.395395 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.395794 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.395907 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:38:42.397048 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.378995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.97.193:39949)
0925 16:38:42.381144 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.381197 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.382247 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.383006 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.383015 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.383636 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.384408 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.384427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385768 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.385771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.387236 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.387247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.387960 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.388276 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.389220 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.389236 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.391426 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.393885 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.393891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.393905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.394244 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.394570 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.394580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.394745 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.395234 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.395240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.395527 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.395731 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.396042 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"mutex_wait_us":475,"spinlock_wait_cycles":35328,"thread_start_us":695,"threads_started":1}
I20260925 16:38:42.397303 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.379241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34283 (local address 127.0.97.254:37533)
0925 16:38:42.379400 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.379406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.381328 (+ 1922us) server_negotiation.cc:407] Connection header received
0925 16:38:42.382489 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.382493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.382548 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.382606 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.384577 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385413 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387374 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.387709 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.391982 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.391987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.391993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.392028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.394832 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.395444 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.395450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.395454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.395550 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.395914 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.395920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.395924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.396170 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.396254 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.396924 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.397060 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:38:42.397318 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.380336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.97.193:49623)
0925 16:38:42.381741 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.381754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.382267 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.383007 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.383017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.383657 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.384409 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.384428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385578 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.385582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.387258 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.387271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.388117 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.388123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.388269 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.389035 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.389060 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.392210 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.394395 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.394400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.394642 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.394940 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.394945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.395015 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.395473 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.395475 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.395537 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.396257 (+ 720us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.396392 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"mutex_wait_us":211,"spinlock_wait_cycles":33408,"thread_start_us":73,"threads_started":1}
I20260925 16:38:42.397686 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.378681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:39949 (local address 127.0.97.252:35927)
0925 16:38:42.379566 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.379579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.381369 (+ 1790us) server_negotiation.cc:407] Connection header received
0925 16:38:42.382515 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.382518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.382569 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.382640 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.384576 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385651 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387377 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.387720 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.391634 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.391639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.391642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.391672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.393740 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394363 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.394368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.394373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.394463 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.394856 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.394862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.394864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.395099 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.395185 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.397476 (+ 2291us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.397575 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":85,"threads_started":1}
I20260925 16:38:42.398583 7454 heartbeater.cc:347] Connected to a master server at 127.0.97.252:35927
I20260925 16:38:42.398815 7454 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.399294 7454 heartbeater.cc:511] Master 127.0.97.252:35927 requested a full tablet report, sending...
I20260925 16:38:42.399331 7456 heartbeater.cc:347] Connected to a master server at 127.0.97.253:40879
I20260925 16:38:42.399407 7456 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.399570 7456 heartbeater.cc:511] Master 127.0.97.253:40879 requested a full tablet report, sending...
I20260925 16:38:42.400185 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.378995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:34283)
0925 16:38:42.381207 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.381219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.382247 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.383006 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.383015 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.383633 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.384408 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.384427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.385569 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.385572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.387247 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.387253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.387960 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.387964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.388277 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.389095 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.389112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.391816 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.395034 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.395039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.395043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.395311 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.395676 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.395680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.395682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.395778 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.396580 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.396584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.396691 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.399785 (+ 3094us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.399933 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"mutex_wait_us":773,"spinlock_wait_cycles":25088,"thread_start_us":131,"threads_started":1}
I20260925 16:38:42.400718 7455 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:42.400789 7455 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.400784 7092 ts_manager.cc:194] Registered new tserver with Master: 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965)
I20260925 16:38:42.400784 7275 ts_manager.cc:194] Registered new tserver with Master: 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965)
I20260925 16:38:42.400916 7455 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:42.401641 6491 ts_manager.cc:194] Re-registered known tserver with Master: 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965)
I20260925 16:38:42.402398 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:34283
W20260925 16:38:42.504683 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:42.505126 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:42.505223 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:42.510564 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:42.510624 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:42.510717 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:42.516319 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:38797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=32909
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:42.517396 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:42.518401 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:42.528194 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.528194 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.528261 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.528273 7462 server_base.cc:1034] running on GCE node
I20260925 16:38:42.528931 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:42.529698 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:42.530885 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790354322530870 us; error 27 us; skew 500 ppm
I20260925 16:38:42.531219 6950 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:42.537947 7462 init.cc:377] Logged in from keytab as oryx/127.0.97.194@KRBTEST.COM (short username oryx)
I20260925 16:38:42.538812 7462 webserver.cc:460] Webserver started at http://127.0.97.194:32909/ using document root <none> and password file <none>
I20260925 16:38:42.539390 7462 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.539502 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.543493 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:42.545548 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.546799 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:42.546942 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "ed1e284ddd77479ca69c1288e0c15f42"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "2a3e1089228d472b0278b8016e778892"
server_key_iv: "301494b3d5ab314468a5c351ace5e0a4"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.547505 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.561918 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:42.564862 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:42.565064 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:42.565696 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:42.566563 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:42.566636 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.566708 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:42.566751 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.576164 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:38797
I20260925 16:38:42.576184 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:38797 every 8 connection(s)
I20260925 16:38:42.577195 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:42.583146 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7462
I20260925 16:38:42.583318 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6821
I20260925 16:38:42.590987 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:32799
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:37533
--webserver_port=39583
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:38:42.595708 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.583102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42154)
0925 16:38:42.583815 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.583832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.583915 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.590013 (+ 6098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.590019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.590071 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.590237 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.590248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.593257 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.593262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.594418 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.594428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.595359 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.595363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.595428 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.595448 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.595490 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.595586 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620}
I20260925 16:38:42.596372 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.584375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.0.97.252:35927)
0925 16:38:42.589712 (+ 5337us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.589718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.589735 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:38:42.589788 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.589790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.589855 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.589970 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.590409 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.593102 (+ 2693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.594571 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.595172 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.596038 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.596166 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.596198 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.596255 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5249,"thread_start_us":341,"threads_started":1}
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:42.597237 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 16:38:42.597929 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897240c44b18481e87b83a49e1cfd688. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.598085 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: LEARNER
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
I20260925 16:38:42.606412 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.578911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37975 (local address 127.0.97.253:40879)
0925 16:38:42.579039 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.579043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.580309 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 16:38:42.581193 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.581197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.581254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.581338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.583868 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.586139 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.588224 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.588572 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.588665 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.598124 (+ 9459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.598130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.598135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.598175 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.603277 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.604018 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.604024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.604027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.604133 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.604617 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.604622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.604626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.604905 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.604988 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.606091 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.606230 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:38:42.607219 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.580584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:55285 (local address 127.0.97.252:35927)
0925 16:38:42.580709 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.580714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.580759 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:38:42.581170 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.581174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.581233 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.581350 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.588000 (+ 6650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.588983 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.593977 (+ 4994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.594318 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.594423 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.603380 (+ 8957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.603385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.603388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.603422 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.605517 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.605998 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.606002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.606003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.606079 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.606360 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.606363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.606365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.606557 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.606620 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.606955 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.607078 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:38:42.608578 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.580450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.97.194:55285)
0925 16:38:42.580685 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.580698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.580954 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.581540 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.581547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.582254 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.585025 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.585055 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.589113 (+ 4058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.589116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.593834 (+ 4718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.593850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.594578 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.594582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.594697 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.600690 (+ 5993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.600708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.603204 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.605641 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.605645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.605649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.605888 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.606182 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.606185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.606187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.606266 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.606686 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.606689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.606774 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.607394 (+ 620us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.607537 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260925 16:38:42.608583 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.579366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.97.194:37975)
0925 16:38:42.580211 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.580224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.580954 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.581540 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.581547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.583260 (+ 1713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.583689 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.583710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.586349 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.586353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.588088 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.588098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.589007 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.589013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.589249 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.594821 (+ 5572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.594849 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.597870 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.603437 (+ 5567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.603447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.603473 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.603890 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.604269 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.604273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.604275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.604481 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.605073 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.605083 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.605773 (+ 690us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.606775 (+ 1002us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.607180 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":24,"spinlock_wait_cycles":35712,"thread_start_us":152,"threads_started":1}
I20260925 16:38:42.609895 7591 heartbeater.cc:347] Connected to a master server at 127.0.97.252:35927
I20260925 16:38:42.609941 7593 heartbeater.cc:347] Connected to a master server at 127.0.97.253:40879
I20260925 16:38:42.610139 7593 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.610663 7593 heartbeater.cc:511] Master 127.0.97.253:40879 requested a full tablet report, sending...
I20260925 16:38:42.610139 7591 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.611064 7591 heartbeater.cc:511] Master 127.0.97.252:35927 requested a full tablet report, sending...
I20260925 16:38:42.611912 7275 ts_manager.cc:194] Registered new tserver with Master: ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:42.611923 7092 ts_manager.cc:194] Registered new tserver with Master: ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:42.613320 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.578975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:45797 (local address 127.0.97.254:37533)
0925 16:38:42.579098 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.579103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.580036 (+ 933us) server_negotiation.cc:407] Connection header received
0925 16:38:42.581183 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.581186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.581233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.581289 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.585730 (+ 4441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.586644 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.588299 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.588767 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.588855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.600650 (+ 11795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.600655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.600659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.600693 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.608498 (+ 7805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.609021 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.609027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.609030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.609120 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.609552 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.609556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.609557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.609769 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.609870 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.613017 (+ 3147us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.613146 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:42.613817 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.579366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:45797)
0925 16:38:42.579862 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.579911 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.580947 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.581538 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.581545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.582190 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.585589 (+ 3399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.585600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.586787 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.586793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.588157 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.588171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.589025 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.589031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.589253 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.598652 (+ 9399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.598674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.600501 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.608627 (+ 8126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.608633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.608637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.608899 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.609236 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.609242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.609265 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.609435 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.612717 (+ 3282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.612720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.612822 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.613494 (+ 672us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.613635 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":198,"thread_start_us":646,"threads_started":1}
I20260925 16:38:42.614442 7592 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:42.614526 7592 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.614682 7592 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:42.615397 6491 ts_manager.cc:194] Re-registered known tserver with Master: ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:42.616272 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:45797
W20260925 16:38:42.716128 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:42.716390 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:42.716431 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:42.720383 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:42.720444 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:42.720502 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:42.725632 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=39583
--tserver_master_addrs=127.0.97.254:37533,127.0.97.253:40879,127.0.97.252:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:42.726635 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:42.727712 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:42.737493 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.737514 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:42.737493 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:42.737835 7599 server_base.cc:1034] running on GCE node
I20260925 16:38:42.738179 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:42.738775 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:42.739961 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790354322739928 us; error 52 us; skew 500 ppm
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:42.742668 7599 init.cc:377] Logged in from keytab as oryx/127.0.97.195@KRBTEST.COM (short username oryx)
I20260925 16:38:42.743502 7599 webserver.cc:460] Webserver started at http://127.0.97.195:39583/ using document root <none> and password file <none>
I20260925 16:38:42.744055 7599 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:42.744155 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:42.747969 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 16:38:42.749971 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.751255 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:42.751408 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
format_stamp: "Formatted at 2026-09-25 16:38:41 on dist-test-slave-kxl8"
server_key: "8e2648a21186e26785ca6e2056addd33"
server_key_iv: "aa752ff1de0647e8e3e6556c1cddd390"
server_key_version: "encryptionkey@0"
I20260925 16:38:42.751997 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:42.773329 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:42.776814 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:42.777033 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:42.777712 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:42.778579 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:42.778666 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.778749 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:42.778802 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:42.787583 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:32799
I20260925 16:38:42.787650 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:32799 every 8 connection(s)
I20260925 16:38:42.788605 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:42.789054 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7599
I20260925 16:38:42.802443 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.791391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.0.97.254:37533)
0925 16:38:42.791735 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.791746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.791977 (+ 231us) server_negotiation.cc:407] Connection header received
0925 16:38:42.792186 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.792190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.792247 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.792332 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.793496 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.794549 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.795785 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.796107 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.796204 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.798052 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.798055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.798058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.798083 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.800269 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.800728 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.800731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.800733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.800815 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.801140 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.801143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.801145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.801384 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.801510 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.802116 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.802239 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":79,"threads_started":1}
I20260925 16:38:42.804618 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.792000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.0.97.252:35927)
0925 16:38:42.792120 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.792125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.794564 (+ 2439us) server_negotiation.cc:407] Connection header received
0925 16:38:42.794734 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.794737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.794790 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.794848 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.795737 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.796468 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.797560 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.797883 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.797951 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.800159 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.800162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.800165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.800190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.802214 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.802974 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.802979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.802980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.803066 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.803563 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.803568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.803571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.803771 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.803875 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.804271 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.804407 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:38:42.804941 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.791282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.97.253:40879)
0925 16:38:42.791402 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.791406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.794261 (+ 2855us) server_negotiation.cc:407] Connection header received
0925 16:38:42.794382 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.794386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.794439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.794514 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.795449 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.796235 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.797400 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.797710 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.797776 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.799521 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.799525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.799529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.799560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.801929 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.803334 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.803337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.803339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.803418 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.803744 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.803747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.803749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.803966 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.804073 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.804635 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.804789 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.252@KRBTEST.COM
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:42.811028 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.791099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:44757 (local address 127.0.97.252:35927)
0925 16:38:42.791233 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.791239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.793706 (+ 2467us) server_negotiation.cc:407] Connection header received
0925 16:38:42.795022 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.795025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.795073 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.795124 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.800099 (+ 4975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.800850 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.802557 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.802931 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.803024 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.806796 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.806800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.806802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.806824 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.808461 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.809099 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.809103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.809105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.809189 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.809711 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.809717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.809718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.809928 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.810024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.810768 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.810884 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:38:42.812126 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.790866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:60573 (local address 127.0.97.254:37533)
0925 16:38:42.791006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.791011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.793679 (+ 2668us) server_negotiation.cc:407] Connection header received
0925 16:38:42.796753 (+ 3074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.796757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.796819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.796885 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.797547 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.798292 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.800042 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.800384 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.800497 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.807629 (+ 7132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.807633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.807636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.807659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.809396 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.809922 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.809925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.809927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.810002 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.810266 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.810269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.810271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.810479 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.810566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.811882 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.811992 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:38:42.812408 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.790972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.97.195:44757)
0925 16:38:42.793537 (+ 2565us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.793649 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.794913 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.795432 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.795440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.795927 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.796583 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.796610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.800989 (+ 4379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.800992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.801864 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.801874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.803139 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.803142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.803251 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.804030 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.804045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.806639 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.808580 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.808586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.808603 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.808943 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.809424 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.809428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.809430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.809596 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.810127 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.810133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.810439 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.811158 (+ 719us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.811317 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2421,"mutex_wait_us":526,"spinlock_wait_cycles":55552,"thread_start_us":656,"threads_started":1}
Sep 25 16:38:42 dist-test-slave-kxl8 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354322, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.253@KRBTEST.COM
I20260925 16:38:42.812917 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.790972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:60573)
0925 16:38:42.793538 (+ 2566us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.793593 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.794572 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.797037 (+ 2465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.797040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.797242 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.797400 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.797407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.798462 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.798467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.799929 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.799940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.802030 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.802037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.802251 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.802780 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.802800 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.807468 (+ 4668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.809543 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.809547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.809549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.809815 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.810095 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.810098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.810100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.810176 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.810658 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.810661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.810778 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.810896 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.811254 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2330,"spinlock_wait_cycles":36096,"thread_start_us":1147,"threads_started":1}
I20260925 16:38:42.814106 7728 heartbeater.cc:347] Connected to a master server at 127.0.97.252:35927
I20260925 16:38:42.814103 7729 heartbeater.cc:347] Connected to a master server at 127.0.97.254:37533
I20260925 16:38:42.814383 7729 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.814383 7728 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.815013 7729 heartbeater.cc:511] Master 127.0.97.254:37533 requested a full tablet report, sending...
I20260925 16:38:42.815018 7728 heartbeater.cc:511] Master 127.0.97.252:35927 requested a full tablet report, sending...
I20260925 16:38:42.816171 6490 ts_manager.cc:194] Re-registered known tserver with Master: 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
I20260925 16:38:42.816972 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:60573
I20260925 16:38:42.817402 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.793056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.97.195:33237)
0925 16:38:42.793580 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.793593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.794746 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.808235 (+ 13489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.808239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.808432 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:42.808573 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.808580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.809730 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.809732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.810594 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.810602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.811311 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.811315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.811404 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.811803 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:42.811816 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:42.813186 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:42.815588 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.815591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.815593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.815865 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.816214 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:42.816218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:42.816220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:42.816322 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:42.816824 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:42.816828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:42.816923 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.817030 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.817170 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"spinlock_wait_cycles":40832,"thread_start_us":99,"threads_started":1}
I20260925 16:38:42.817431 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.793206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:33237 (local address 127.0.97.253:40879)
0925 16:38:42.807994 (+ 14788us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.808000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.808012 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:42.808046 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.808049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.808093 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.808692 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.808787 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.809617 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.810744 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.811168 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.811593 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.813376 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.813381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.813383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.813407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.815450 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.815996 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.816000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.816002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.816093 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.816433 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.816437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.816439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.816705 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.816798 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.817103 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.817239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9447,"thread_start_us":105,"threads_started":1}
I20260925 16:38:42.817582 7275 ts_manager.cc:194] Registered new tserver with Master: 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
I20260925 16:38:42.807850 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:42.818507 7730 heartbeater.cc:347] Connected to a master server at 127.0.97.253:40879
I20260925 16:38:42.818619 7730 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:42.818826 7730 heartbeater.cc:511] Master 127.0.97.253:40879 requested a full tablet report, sending...
I20260925 16:38:42.819582 7092 ts_manager.cc:194] Registered new tserver with Master: 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
W20260925 16:38:42.820139 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:42.822932 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: NON_VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: true } } }
I20260925 16:38:42.823056 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.826351 7745 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: attempting to promote NON_VOTER b7ae709c2e094ba7a4a4160da30206e7 to VOTER
I20260925 16:38:42.826702 7745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } }
I20260925 16:38:42.828027 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 LEARNER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:38:42.828027 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Refusing update from remote peer 897240c44b18481e87b83a49e1cfd688: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:38:42.828410 7746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 16:38:42.828564 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 16:38:42.830909 7745 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.831857 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.831753 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.832445 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.832551 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.832645 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.832705 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.832621 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.832830 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:42.833984 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.834113 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:42.834607 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "897240c44b18481e87b83a49e1cfd688" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:42.834738 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:42.842278 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.836552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34965 (local address 127.0.0.1:48856)
0925 16:38:42.836706 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.836722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.836800 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.838240 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.838243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.838259 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.838391 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.838398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.839969 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.839972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.840958 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.840966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.841916 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.841920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.841999 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.842020 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.842081 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.842150 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":42}
I20260925 16:38:42.842367 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.836751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.0.97.193:34965)
0925 16:38:42.837100 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.837116 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.837635 (+ 519us) server_negotiation.cc:407] Connection header received
0925 16:38:42.837792 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.837798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.838055 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.838283 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.838484 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.839833 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.841062 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.841796 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.841921 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.842076 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.842127 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.842217 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":152,"threads_started":1}
I20260925 16:38:42.843012 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.837083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38797 (local address 127.0.0.1:48560)
0925 16:38:42.837575 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.837586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.837690 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.838599 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.838602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.838621 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.838706 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.838711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.840121 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.840126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.841378 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.841387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.842716 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.842718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.842781 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.842798 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.842864 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.842911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":2,"thread_start_us":130,"threads_started":1}
I20260925 16:38:42.843251 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.837258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.0.97.194:38797)
0925 16:38:42.837953 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.837963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.837981 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:38:42.838078 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.838083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.838287 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.838436 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.838808 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.839995 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.841563 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.842592 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.842723 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.842853 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.842900 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.842988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":124,"threads_started":1}
I20260925 16:38:42.845036 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.837332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:32799 (local address 127.0.0.1:34996)
0925 16:38:42.838496 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.838510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.838617 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.839031 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.839034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.839050 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.839130 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.839136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.841343 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.841347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.843479 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.843490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.844647 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.844651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.844739 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.844759 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.844826 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.844889 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":92,"threads_started":1}
I20260925 16:38:42.845424 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.837857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.97.195:32799)
0925 16:38:42.838168 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.838178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.838608 (+ 430us) server_negotiation.cc:407] Connection header received
0925 16:38:42.838722 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.838728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.838904 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.839052 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.839219 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.839984 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.843603 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.844360 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.844471 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.844587 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.845175 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.845266 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260925 16:38:42.845570 7388 tablet_service.cc:1467] Processing CreateTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e (DEFAULT_TABLE table=test-table [id=2bf623817acb4cdb84e518f0bb163da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:42.846661 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9bd0f1f54642da83bcdb28007ed43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:42.846211 7525 tablet_service.cc:1467] Processing CreateTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e (DEFAULT_TABLE table=test-table [id=2bf623817acb4cdb84e518f0bb163da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:42.847306 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9bd0f1f54642da83bcdb28007ed43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:42.848234 7662 tablet_service.cc:1467] Processing CreateTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e (DEFAULT_TABLE table=test-table [id=2bf623817acb4cdb84e518f0bb163da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:42.849306 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9bd0f1f54642da83bcdb28007ed43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:42.853309 7760 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Bootstrap starting.
I20260925 16:38:42.855425 7760 tablet_bootstrap.cc:654] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:42.856161 7760 log.cc:824] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:42.856585 7761 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Bootstrap starting.
I20260925 16:38:42.858706 7760 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: No bootstrap required, opened a new log
I20260925 16:38:42.858692 7763 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Bootstrap starting.
I20260925 16:38:42.858855 7761 tablet_bootstrap.cc:654] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:42.858896 7760 ts_tablet_manager.cc:1397] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:38:42.859743 7761 log.cc:824] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:42.860634 7763 tablet_bootstrap.cc:654] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:42.861738 7761 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: No bootstrap required, opened a new log
I20260925 16:38:42.861831 7763 log.cc:824] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:42.861886 7761 ts_tablet_manager.cc:1397] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:38:42.863780 7763 tablet_bootstrap.cc:492] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: No bootstrap required, opened a new log
I20260925 16:38:42.863692 7760 raft_consensus.cc:348] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.863938 7763 ts_tablet_manager.cc:1397] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:38:42.863967 7760 raft_consensus.cc:374] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:42.864012 7760 raft_consensus.cc:729] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ab6503654145068332e88f3ea709ac, State: Initialized, Role: FOLLOWER
I20260925 16:38:42.864430 7760 consensus_queue.cc:260] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.865439 7760 ts_tablet_manager.cc:1428] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:38:42.865882 7455 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
I20260925 16:38:42.865986 7761 raft_consensus.cc:348] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.866232 7761 raft_consensus.cc:374] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:42.866284 7761 raft_consensus.cc:729] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e284ddd77479ca69c1288e0c15f42, State: Initialized, Role: FOLLOWER
I20260925 16:38:42.866766 7761 consensus_queue.cc:260] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.867782 7761 ts_tablet_manager.cc:1428] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:38:42.868275 7763 raft_consensus.cc:348] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.868528 7763 raft_consensus.cc:374] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:42.868592 7763 raft_consensus.cc:729] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc9b331c63641e6be7aa44d07d8fe46, State: Initialized, Role: FOLLOWER
I20260925 16:38:42.868999 7763 consensus_queue.cc:260] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.869159 7592 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
I20260925 16:38:42.870222 7763 ts_tablet_manager.cc:1428] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:38:42.871176 7768 raft_consensus.cc:482] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:42.871323 7768 raft_consensus.cc:504] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.871395 7729 heartbeater.cc:503] Master 127.0.97.254:37533 was elected leader, sending a full tablet report...
I20260925 16:38:42.874800 7768 leader_election.cc:288] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799), ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:42.877460 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.873487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:58663 (local address 127.0.97.195:32799)
0925 16:38:42.873597 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.873601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.873612 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:38:42.873650 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.873653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.873693 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.873760 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.874145 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.874891 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.876312 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.876873 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.876986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.877024 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.877235 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.877351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:38:42.877467 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.873232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:32799 (local address 127.0.97.193:58663)
0925 16:38:42.873386 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.873422 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.873502 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.873869 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.873874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.873895 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.874015 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.874022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.875013 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.875017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.876185 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.876196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.876993 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.876997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.877100 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.877129 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.877209 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.877334 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 16:38:42.877935 7457 tablet.cc:2369] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:42.877981 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9bd0f1f54642da83bcdb28007ed43e" candidate_uuid: "60ab6503654145068332e88f3ea709ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" is_pre_election: true
I20260925 16:38:42.878243 7682 raft_consensus.cc:2454] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ab6503654145068332e88f3ea709ac in term 0.
I20260925 16:38:42.878728 7344 leader_election.cc:302] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ab6503654145068332e88f3ea709ac, 7fc9b331c63641e6be7aa44d07d8fe46; no voters:
I20260925 16:38:42.879014 7768 raft_consensus.cc:2781] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:42.879105 7768 raft_consensus.cc:482] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:42.879163 7768 raft_consensus.cc:3037] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:42.879307 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.875032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38797 (local address 127.0.97.193:42197)
0925 16:38:42.875167 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:42.875179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:42.875241 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:42.875571 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:42.875574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:42.875593 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:42.875691 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.875698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.876853 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.876856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.878108 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:42.878119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.878912 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.878916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:42.879012 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.879033 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:42.879125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:38:42.879192 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 16:38:42.879307 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.875088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:42197 (local address 127.0.97.194:38797)
0925 16:38:42.875210 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.875215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.875230 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:42.875324 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.875327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.875382 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.875459 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:42.875944 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.876721 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.878257 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.878810 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.878923 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.878968 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.879138 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.879199 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:38:42.879908 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9bd0f1f54642da83bcdb28007ed43e" candidate_uuid: "60ab6503654145068332e88f3ea709ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e284ddd77479ca69c1288e0c15f42" is_pre_election: true
I20260925 16:38:42.880119 7545 raft_consensus.cc:2454] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ab6503654145068332e88f3ea709ac in term 0.
I20260925 16:38:42.880705 7768 raft_consensus.cc:504] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.881099 7768 leader_election.cc:288] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [CANDIDATE]: Term 1 election: Requested vote from peers 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799), ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797)
I20260925 16:38:42.881397 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9bd0f1f54642da83bcdb28007ed43e" candidate_uuid: "60ab6503654145068332e88f3ea709ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
I20260925 16:38:42.881457 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9bd0f1f54642da83bcdb28007ed43e" candidate_uuid: "60ab6503654145068332e88f3ea709ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e284ddd77479ca69c1288e0c15f42"
I20260925 16:38:42.881526 7682 raft_consensus.cc:3037] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:42.881593 7545 raft_consensus.cc:3037] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:42.882618 7545 raft_consensus.cc:2454] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ab6503654145068332e88f3ea709ac in term 1.
I20260925 16:38:42.882620 7682 raft_consensus.cc:2454] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ab6503654145068332e88f3ea709ac in term 1.
I20260925 16:38:42.882884 7341 leader_election.cc:302] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ab6503654145068332e88f3ea709ac, ed1e284ddd77479ca69c1288e0c15f42; no voters:
I20260925 16:38:42.883102 7768 raft_consensus.cc:2781] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:42.883262 7768 raft_consensus.cc:686] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 1 LEADER]: Becoming Leader. State: Replica: 60ab6503654145068332e88f3ea709ac, State: Running, Role: LEADER
I20260925 16:38:42.883742 7768 consensus_queue.cc:237] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } }
I20260925 16:38:42.886976 6491 catalog_manager.cc:5495] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ab6503654145068332e88f3ea709ac (127.0.97.193). New cstate: current_term: 1 leader_uuid: "60ab6503654145068332e88f3ea709ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:42.911520 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.908119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.97.193:34965)
0925 16:38:42.908237 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.908241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.908254 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:42.908336 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.908339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.908392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.908470 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:42.908695 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.909230 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.910205 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.910452 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.910530 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.910798 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:42.911218 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:42.911304 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.911374 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.911421 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:38:42.916971 7545 raft_consensus.cc:1261] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Refusing update from remote peer 60ab6503654145068332e88f3ea709ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:42.916971 7682 raft_consensus.cc:1261] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Refusing update from remote peer 60ab6503654145068332e88f3ea709ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:42.917811 7768 consensus_queue.cc:1035] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:42.918073 7766 consensus_queue.cc:1035] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:42.926851 7774 mvcc.cc:205] Tried to move back new op lower bound from 7333291306660872192 to 7333291306534354944. Current Snapshot: MvccSnapshot[applied={T|T < 7333291306660872192}]
I20260925 16:38:42.940549 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:42.940734 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:42.944878 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897240c44b18481e87b83a49e1cfd688: Sending DeleteTablet for 3 replicas of tablet ed9bd0f1f54642da83bcdb28007ed43e
I20260925 16:38:42.945576 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:42 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:48856
I20260925 16:38:42.946154 7782 tablet_replica.cc:330] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: stopping tablet replica
I20260925 16:38:42.946626 7782 raft_consensus.cc:2229] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:42.946588 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:42 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:48560
I20260925 16:38:42.946599 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9bd0f1f54642da83bcdb28007ed43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:42 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:34996
I20260925 16:38:42.947139 7783 tablet_replica.cc:330] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: stopping tablet replica
I20260925 16:38:42.947261 7782 raft_consensus.cc:2258] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:42.947453 7783 raft_consensus.cc:2229] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:42.947769 7783 raft_consensus.cc:2258] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:42.949031 7783 ts_tablet_manager.cc:1905] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:42.949031 7782 ts_tablet_manager.cc:1905] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:42.950181 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6459
I20260925 16:38:42.952970 7783 ts_tablet_manager.cc:1918] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:42.953133 7783 log.cc:1196] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/ed9bd0f1f54642da83bcdb28007ed43e
I20260925 16:38:42.953305 7782 ts_tablet_manager.cc:1918] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:42.953483 7782 log.cc:1196] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/ed9bd0f1f54642da83bcdb28007ed43e
I20260925 16:38:42.953500 7783 ts_tablet_manager.cc:1939] T ed9bd0f1f54642da83bcdb28007ed43e P ed1e284ddd77479ca69c1288e0c15f42: Deleting consensus metadata
I20260925 16:38:42.953863 7782 ts_tablet_manager.cc:1939] T ed9bd0f1f54642da83bcdb28007ed43e P 60ab6503654145068332e88f3ea709ac: Deleting consensus metadata
W20260925 16:38:42.957593 7342 connection.cc:535] server connection from 127.0.0.1:48856 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 16:38:42.958031 7479 connection.cc:535] server connection from 127.0.0.1:48560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:38:42.958678 7784 tablet_replica.cc:330] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: stopping tablet replica
I20260925 16:38:42.959089 7784 raft_consensus.cc:2229] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:42.959519 7784 raft_consensus.cc:2258] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:42.961194 7784 ts_tablet_manager.cc:1905] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:42.964324 7784 ts_tablet_manager.cc:1918] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:42.964437 7784 log.cc:1196] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/ed9bd0f1f54642da83bcdb28007ed43e
I20260925 16:38:42.964845 7784 ts_tablet_manager.cc:1939] T ed9bd0f1f54642da83bcdb28007ed43e P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting consensus metadata
W20260925 16:38:42.966106 7615 connection.cc:441] server connection from 127.0.0.1:34996 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34996 (request call id 1) could send its response
I20260925 16:38:42.967667 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.949601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.97.253:40879)
0925 16:38:42.949723 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.949727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.958332 (+ 8605us) server_negotiation.cc:407] Connection header received
0925 16:38:42.958508 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.958511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.958564 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.958644 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.959676 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.960438 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.961573 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.961900 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.961977 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.964044 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.964048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.964051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.964077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.965885 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.966392 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.966395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.966397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.966484 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.966743 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.966748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.966750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.966985 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.967134 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.967388 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.967505 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 16:38:42.975807 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.962100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.0.97.253:40879)
0925 16:38:42.962218 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.962222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.963465 (+ 1243us) server_negotiation.cc:407] Connection header received
0925 16:38:42.963850 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.963854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.963897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.963944 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.966037 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.966818 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.970226 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.970554 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.970621 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.972195 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.972199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.972202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.972240 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.973899 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.974434 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.974438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.974440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.974523 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.974841 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.974844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.974846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.975058 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.975160 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.975523 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.975631 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:38:42.976531 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.962682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.97.252:35927)
0925 16:38:42.962793 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.962797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.963678 (+ 881us) server_negotiation.cc:407] Connection header received
0925 16:38:42.964700 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.964704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.964778 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.964831 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.967855 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.968570 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.969738 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.970069 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.970165 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.971936 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.971941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.971945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.971971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.974137 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.974656 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.974663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.974666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.974757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.975081 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.975085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.975088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.975463 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.975563 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.976220 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.976340 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:38:42.977630 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:42.949714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.97.252:35927)
0925 16:38:42.949813 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:42.949817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:42.967717 (+ 17900us) server_negotiation.cc:407] Connection header received
0925 16:38:42.968052 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:42.968055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:42.968101 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:42.968196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:42.968834 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.969393 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.970481 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:42.971394 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:42.971481 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:42.973660 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:42.973664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:42.973667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:42.973693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:42.975347 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.976421 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.976424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.976426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.976536 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:42.976806 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:42.976811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:42.976812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:42.976988 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:42.977121 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:42.977366 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 16:38:42.977472 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 16:38:43.930429 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:43.929948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:59827)
0925 16:38:43.930197 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:43.930285 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
W20260925 16:38:43.930943 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:43.931859 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:43.931371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:47121)
0925 16:38:43.931630 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:43.931718 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
W20260925 16:38:43.932969 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:43.933377 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:43.932619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:51411)
0925 16:38:43.933167 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:43.933259 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":74,"threads_started":1}
W20260925 16:38:43.933643 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.445389 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 897240c44b18481e87b83a49e1cfd688)
I20260925 16:38:44.445622 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } }
I20260925 16:38:44.446463 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 897240c44b18481e87b83a49e1cfd688)
I20260925 16:38:44.446688 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533), b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252:35927)
I20260925 16:38:44.446733 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } }
I20260925 16:38:44.447587 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.447074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47110)
0925 16:38:44.447402 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.447486 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":60,"threads_started":1}
W20260925 16:38:44.447836 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.448397 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.447993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47124)
0925 16:38:44.448238 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.448302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
I20260925 16:38:44.448467 7797 leader_election.cc:288] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533), 3818a2fff337466fbdc4d305e38beb60 (127.0.97.253:40879)
W20260925 16:38:44.448570 7260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.449044 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.448791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47128)
0925 16:38:44.448926 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.448989 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 16:38:44.449173 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533): Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.449421 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.449183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47134)
0925 16:38:44.449305 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.449363 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 16:38:44.449553 7260 leader_election.cc:334] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533): Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.451318 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.446940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:35927 (local address 127.0.0.1:42194)
0925 16:38:44.447308 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.447324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.447426 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.447834 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.447836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.447865 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.447963 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.447969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.448990 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.448993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.450200 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.450209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.450920 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.450923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.451013 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.451034 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.451130 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.451198 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":80,"threads_started":1}
I20260925 16:38:44.451328 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.447082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.97.252:35927)
0925 16:38:44.447388 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.447392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.447411 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:38:44.447544 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.447547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.447598 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.447695 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.448098 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.448868 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.450310 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.450803 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.450923 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.450989 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.451142 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.451209 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":67,"threads_started":1}
I20260925 16:38:44.451783 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3818a2fff337466fbdc4d305e38beb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7ae709c2e094ba7a4a4160da30206e7" is_pre_election: true
I20260925 16:38:44.451983 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3818a2fff337466fbdc4d305e38beb60 in term 1.
I20260925 16:38:44.452311 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3818a2fff337466fbdc4d305e38beb60, b7ae709c2e094ba7a4a4160da30206e7; no voters: 897240c44b18481e87b83a49e1cfd688
I20260925 16:38:44.452533 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 16:38:44.452586 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 897240c44b18481e87b83a49e1cfd688)
I20260925 16:38:44.452623 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:38:44.452976 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.448289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.0.97.253:40879)
0925 16:38:44.448722 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.448733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.448751 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:38:44.449402 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.449405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.449441 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.449766 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.449982 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.450453 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.451769 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.452353 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.452474 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.452524 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.452816 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.452879 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":93,"threads_started":1}
I20260925 16:38:44.452975 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.448058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40879 (local address 127.0.0.1:49172)
0925 16:38:44.448316 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.448329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.448910 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.449781 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.449785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.449804 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.449883 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.449888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.450567 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.450572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.451640 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.451650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.452500 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.452504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.452597 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.452623 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.452772 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.452844 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":41,"threads_started":1}
I20260925 16:38:44.453466 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7ae709c2e094ba7a4a4160da30206e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3818a2fff337466fbdc4d305e38beb60" is_pre_election: true
I20260925 16:38:44.453969 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } }
I20260925 16:38:44.454202 7102 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7ae709c2e094ba7a4a4160da30206e7 in current term 2: Already voted for candidate 3818a2fff337466fbdc4d305e38beb60 in this term.
I20260925 16:38:44.454357 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 election: Requested vote from peers 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533), b7ae709c2e094ba7a4a4160da30206e7 (127.0.97.252:35927)
I20260925 16:38:44.454656 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.454471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47150)
0925 16:38:44.454556 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.454602 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 16:38:44.454662 7262 leader_election.cc:302] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7ae709c2e094ba7a4a4160da30206e7; no voters: 3818a2fff337466fbdc4d305e38beb60, 897240c44b18481e87b83a49e1cfd688
I20260925 16:38:44.454864 7797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:38:44.455226 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3818a2fff337466fbdc4d305e38beb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7ae709c2e094ba7a4a4160da30206e7"
I20260925 16:38:44.455273 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.455034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47154)
0925 16:38:44.455179 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.455231 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:38:44.455386 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533): Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
I20260925 16:38:44.455946 7797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:38:44.456947 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3818a2fff337466fbdc4d305e38beb60 in term 2.
I20260925 16:38:44.457230 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3818a2fff337466fbdc4d305e38beb60, b7ae709c2e094ba7a4a4160da30206e7; no voters: 897240c44b18481e87b83a49e1cfd688
I20260925 16:38:44.457449 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:38:44.457664 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [term 2 LEADER]: Becoming Leader. State: Replica: 3818a2fff337466fbdc4d305e38beb60, State: Running, Role: LEADER
I20260925 16:38:44.457903 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } }
I20260925 16:38:44.458930 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3818a2fff337466fbdc4d305e38beb60. Latest consensus state: current_term: 2 leader_uuid: "3818a2fff337466fbdc4d305e38beb60" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:44.459028 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:44.459277 7808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:44.461443 7808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:44.461822 7808 catalog_manager.cc:1259] Loaded cluster ID: e78cbf005d91467baf68de4720ddd273
I20260925 16:38:44.461897 7808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:44.462414 7808 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:44.462803 7808 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Loaded TSK: 0
I20260925 16:38:44.463455 7808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:38:44.486675 7092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:44.489771 7092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:44.492789 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.492468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47178)
0925 16:38:44.492616 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.492684 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 16:38:44.492852 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [term 2 FOLLOWER]: Refusing update from remote peer 3818a2fff337466fbdc4d305e38beb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 16:38:44.493415 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 16:38:44.493501 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.493301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.0.1:47192)
0925 16:38:44.493400 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.493455 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 16:38:44.493736 7077 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 -> Peer 897240c44b18481e87b83a49e1cfd688 (127.0.97.254:37533): Couldn't send request to peer 897240c44b18481e87b83a49e1cfd688. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:38:44.494891 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3818a2fff337466fbdc4d305e38beb60. Latest consensus state: current_term: 2 leader_uuid: "3818a2fff337466fbdc4d305e38beb60" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:44.495007 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:44.496129 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3818a2fff337466fbdc4d305e38beb60" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:44.496230 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:44.496371 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3818a2fff337466fbdc4d305e38beb60" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "897240c44b18481e87b83a49e1cfd688" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 37533 } } peers { permanent_uuid: "3818a2fff337466fbdc4d305e38beb60" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40879 } attrs { promote: false } } peers { permanent_uuid: "b7ae709c2e094ba7a4a4160da30206e7" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 35927 } attrs { promote: false } } }
I20260925 16:38:44.496469 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ae709c2e094ba7a4a4160da30206e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:38:44.507640 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.503556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.97.195:32799)
0925 16:38:44.503773 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.503778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.503790 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:44.503849 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.503855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.503910 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.504110 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.504300 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.504820 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.506217 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.506865 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.507019 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.507088 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.507381 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.507463 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260925 16:38:44.507640 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.503174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:32799 (local address 127.0.0.1:35010)
0925 16:38:44.503515 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.503529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.503585 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.504027 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.504029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.504044 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.504185 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.504193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.504933 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.504939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.506064 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.506077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.507054 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.507060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.507188 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.507234 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.507339 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.507438 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":46,"thread_start_us":114,"threads_started":1}
I20260925 16:38:44.508874 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 0345b30f7cf444a48d8b7431e044618a (DEFAULT_TABLE table=test-table [id=afd83e24d4d441c198a5e724461c0fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:44.509227 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345b30f7cf444a48d8b7431e044618a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:44.510586 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.502830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:38797 (local address 127.0.0.1:48562)
0925 16:38:44.503406 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.503418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.503500 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.503942 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.503948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.503973 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.504129 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.504136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.505322 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.505325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.508912 (+ 3587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.508922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.510136 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.510139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.510252 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.510273 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.510371 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.510452 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":1}
I20260925 16:38:44.510587 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.503140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.97.194:38797)
0925 16:38:44.503699 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.503703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.503717 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:44.503766 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.503770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.503820 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.503904 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.504290 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.505164 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.509141 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.509987 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.510142 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.510214 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.510387 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.510462 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":112,"threads_started":1}
I20260925 16:38:44.511664 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 0345b30f7cf444a48d8b7431e044618a (DEFAULT_TABLE table=test-table [id=afd83e24d4d441c198a5e724461c0fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:44.511961 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345b30f7cf444a48d8b7431e044618a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:44.513334 7819 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Bootstrap starting.
I20260925 16:38:44.514519 7820 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Bootstrap starting.
I20260925 16:38:44.515530 7819 tablet_bootstrap.cc:654] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:44.515535 7820 tablet_bootstrap.cc:654] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:44.516589 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.502306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34965 (local address 127.0.0.1:48884)
0925 16:38:44.502955 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.502972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.503061 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.511374 (+ 8313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.511377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.511399 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.511511 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.511519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.512795 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.512798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.514014 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.514027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.515849 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.515856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.515996 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.516038 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.516095 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.516188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582}
I20260925 16:38:44.517230 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.502546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.0.97.193:34965)
0925 16:38:44.511107 (+ 8561us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.511112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.511128 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:38:44.511179 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.511181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.511233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.511635 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.511760 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.512630 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.514373 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.514969 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.516819 (+ 1850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.516901 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.516953 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.517063 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8466,"thread_start_us":74,"threads_started":1}
I20260925 16:38:44.517778 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 0345b30f7cf444a48d8b7431e044618a (DEFAULT_TABLE table=test-table [id=afd83e24d4d441c198a5e724461c0fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:44.518038 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345b30f7cf444a48d8b7431e044618a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:44.518258 7820 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: No bootstrap required, opened a new log
I20260925 16:38:44.518335 7820 ts_tablet_manager.cc:1397] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:38:44.518702 7820 raft_consensus.cc:348] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.519198 7820 raft_consensus.cc:374] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:44.519279 7820 raft_consensus.cc:729] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e284ddd77479ca69c1288e0c15f42, State: Initialized, Role: FOLLOWER
I20260925 16:38:44.519405 7820 consensus_queue.cc:260] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.519762 7820 ts_tablet_manager.cc:1428] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:44.520107 7819 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: No bootstrap required, opened a new log
I20260925 16:38:44.520179 7819 ts_tablet_manager.cc:1397] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 16:38:44.520810 7824 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: Bootstrap starting.
I20260925 16:38:44.520871 7819 raft_consensus.cc:348] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.520987 7819 raft_consensus.cc:374] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:44.521023 7819 raft_consensus.cc:729] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc9b331c63641e6be7aa44d07d8fe46, State: Initialized, Role: FOLLOWER
I20260925 16:38:44.521144 7819 consensus_queue.cc:260] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.522145 7819 ts_tablet_manager.cc:1428] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:44.524422 7824 tablet_bootstrap.cc:654] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:44.525103 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.520758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:57447)
0925 16:38:44.524927 (+ 4169us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.524993 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4087,"thread_start_us":97,"threads_started":1}
I20260925 16:38:44.525306 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.523766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:55285)
0925 16:38:44.525048 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.525095 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":93,"threads_started":1}
I20260925 16:38:44.525466 7730 heartbeater.cc:503] Master 127.0.97.253:40879 was elected leader, sending a full tablet report...
I20260925 16:38:44.526265 7824 tablet_bootstrap.cc:492] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: No bootstrap required, opened a new log
I20260925 16:38:44.526422 7824 ts_tablet_manager.cc:1397] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 16:38:44.526480 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.526112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:57239)
0925 16:38:44.526365 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.526413 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260925 16:38:44.526952 7824 raft_consensus.cc:348] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.527071 7824 raft_consensus.cc:374] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:44.527101 7824 raft_consensus.cc:729] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ab6503654145068332e88f3ea709ac, State: Initialized, Role: FOLLOWER
I20260925 16:38:44.527226 7824 consensus_queue.cc:260] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.527581 7824 ts_tablet_manager.cc:1428] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:44.529193 7456 heartbeater.cc:503] Master 127.0.97.253:40879 was elected leader, sending a full tablet report...
I20260925 16:38:44.530954 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.527953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:36729)
0925 16:38:44.530837 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.530885 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2817,"thread_start_us":140,"threads_started":1}
I20260925 16:38:44.531630 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.531406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:38503)
0925 16:38:44.531516 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.531560 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:38:44.542896 7731 tablet.cc:2369] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:38:44.580468 7594 tablet.cc:2369] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:44.710361 7823 raft_consensus.cc:482] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:44.710512 7823 raft_consensus.cc:504] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.711635 7823 leader_election.cc:288] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965), 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
I20260925 16:38:44.717134 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.712041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:43141 (local address 127.0.97.195:32799)
0925 16:38:44.712205 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.712212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.712342 (+ 130us) server_negotiation.cc:407] Connection header received
0925 16:38:44.712436 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.712438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.712484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.712564 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.712976 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.713797 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.715788 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.716419 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.716546 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.716598 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.716938 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.717012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:38:44.717134 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.712060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:52347 (local address 127.0.97.193:34965)
0925 16:38:44.712329 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.712335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.712350 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:44.712407 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.712411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.712465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.712555 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:44.712970 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.713789 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.715837 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.716429 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.716546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.716598 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.716938 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.717012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 16:38:44.717139 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.711915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34965 (local address 127.0.97.194:52347)
0925 16:38:44.712145 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.712162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.712270 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.712591 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.712594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.712624 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.712861 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.712867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.713921 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.713925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.715647 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.715658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.716551 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.716554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.716744 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.716790 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.716868 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.716949 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":303,"spinlock_wait_cycles":8192}
I20260925 16:38:44.717146 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.711951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:32799 (local address 127.0.97.194:43141)
0925 16:38:44.712241 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.712252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:44.712353 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:44.712591 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:44.712594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:44.712622 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:44.712851 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.712859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.713921 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.713925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.715647 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:44.715658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.716549 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.716553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:44.716723 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.716785 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 16:38:44.716887 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:38:44.716949 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":401,"spinlock_wait_cycles":4224,"thread_start_us":113,"threads_started":1}
I20260925 16:38:44.717710 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345b30f7cf444a48d8b7431e044618a" candidate_uuid: "ed1e284ddd77479ca69c1288e0c15f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ab6503654145068332e88f3ea709ac" is_pre_election: true
I20260925 16:38:44.717888 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345b30f7cf444a48d8b7431e044618a" candidate_uuid: "ed1e284ddd77479ca69c1288e0c15f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" is_pre_election: true
I20260925 16:38:44.718025 7408 raft_consensus.cc:2454] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e284ddd77479ca69c1288e0c15f42 in term 0.
I20260925 16:38:44.718070 7682 raft_consensus.cc:2454] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e284ddd77479ca69c1288e0c15f42 in term 0.
I20260925 16:38:44.718466 7478 leader_election.cc:302] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ab6503654145068332e88f3ea709ac, ed1e284ddd77479ca69c1288e0c15f42; no voters:
I20260925 16:38:44.718729 7823 raft_consensus.cc:2781] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:44.718792 7823 raft_consensus.cc:482] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:44.718829 7823 raft_consensus.cc:3037] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:44.719875 7823 raft_consensus.cc:504] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.720214 7823 leader_election.cc:288] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [CANDIDATE]: Term 1 election: Requested vote from peers 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965), 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799)
I20260925 16:38:44.720510 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345b30f7cf444a48d8b7431e044618a" candidate_uuid: "ed1e284ddd77479ca69c1288e0c15f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ab6503654145068332e88f3ea709ac"
I20260925 16:38:44.720630 7408 raft_consensus.cc:3037] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:44.720630 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345b30f7cf444a48d8b7431e044618a" candidate_uuid: "ed1e284ddd77479ca69c1288e0c15f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc9b331c63641e6be7aa44d07d8fe46"
I20260925 16:38:44.720749 7682 raft_consensus.cc:3037] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:44.721673 7408 raft_consensus.cc:2454] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e284ddd77479ca69c1288e0c15f42 in term 1.
I20260925 16:38:44.721800 7682 raft_consensus.cc:2454] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e284ddd77479ca69c1288e0c15f42 in term 1.
I20260925 16:38:44.721925 7478 leader_election.cc:302] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ab6503654145068332e88f3ea709ac, ed1e284ddd77479ca69c1288e0c15f42; no voters:
I20260925 16:38:44.722105 7823 raft_consensus.cc:2781] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:44.722322 7823 raft_consensus.cc:686] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 1 LEADER]: Becoming Leader. State: Replica: ed1e284ddd77479ca69c1288e0c15f42, State: Running, Role: LEADER
I20260925 16:38:44.722648 7593 heartbeater.cc:503] Master 127.0.97.253:40879 was elected leader, sending a full tablet report...
I20260925 16:38:44.722911 7823 consensus_queue.cc:237] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } }
I20260925 16:38:44.723086 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.722814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:42809)
0925 16:38:44.722926 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.722979 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 16:38:44.725394 7091 catalog_manager.cc:5495] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "ed1e284ddd77479ca69c1288e0c15f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1e284ddd77479ca69c1288e0c15f42" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:44.752974 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.749472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.194:38797)
0925 16:38:44.749593 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:44.749596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:44.749609 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:44.749698 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:44.749701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:44.749756 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:44.749841 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:44.750058 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.750634 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.751461 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:44.751757 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:44.751837 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:44.752117 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:44.752569 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:44.752695 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:44.752777 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:38:44.752830 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:38:44.755566 7682 raft_consensus.cc:1261] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Refusing update from remote peer ed1e284ddd77479ca69c1288e0c15f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:44.755787 7408 raft_consensus.cc:1261] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Refusing update from remote peer ed1e284ddd77479ca69c1288e0c15f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:44.756072 7832 consensus_queue.cc:1035] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fc9b331c63641e6be7aa44d07d8fe46" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:44.756332 7823 consensus_queue.cc:1035] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ab6503654145068332e88f3ea709ac" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:44.758172 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.757813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:56191)
0925 16:38:44.758024 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.758092 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 16:38:44.758761 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.758376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:46501)
0925 16:38:44.758629 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.758688 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260925 16:38:44.760452 7835 mvcc.cc:205] Tried to move back new op lower bound from 7333291314194038784 to 7333291314067521536. Current Snapshot: MvccSnapshot[applied={T|T < 7333291314194038784}]
I20260925 16:38:44.761305 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.760924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:57313)
0925 16:38:44.761156 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.761211 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 16:38:44.761938 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.761721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.195:47025)
0925 16:38:44.761819 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.761870 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 16:38:44.763019 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.762779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.194:40177)
0925 16:38:44.762917 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.762964 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 16:38:44.763598 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.763354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:37533 (local address 127.0.97.193:59491)
0925 16:38:44.763489 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.763533 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 16:38:44.764170 7837 mvcc.cc:205] Tried to move back new op lower bound from 7333291314194038784 to 7333291314067521536. Current Snapshot: MvccSnapshot[applied={T|T < 7333291314194038784}]
I20260925 16:38:44.770614 7092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:44.770740 7092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:44.774530 7092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3818a2fff337466fbdc4d305e38beb60: Sending DeleteTablet for 3 replicas of tablet 0345b30f7cf444a48d8b7431e044618a
I20260925 16:38:44.775180 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345b30f7cf444a48d8b7431e044618a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:44 UTC) from {username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:48884
I20260925 16:38:44.775242 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345b30f7cf444a48d8b7431e044618a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:44 UTC) from {username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:48562
I20260925 16:38:44.775574 7846 tablet_replica.cc:330] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: stopping tablet replica
I20260925 16:38:44.775580 7845 tablet_replica.cc:330] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac: stopping tablet replica
I20260925 16:38:44.775701 7846 raft_consensus.cc:2229] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:44.775698 7845 raft_consensus.cc:2229] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:44.775825 7845 raft_consensus.cc:2258] T 0345b30f7cf444a48d8b7431e044618a P 60ab6503654145068332e88f3ea709ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:44.775985 7846 raft_consensus.cc:2258] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:44.776329 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7237
I20260925 16:38:44.776419 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345b30f7cf444a48d8b7431e044618a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:44 UTC) from {username='oryx', principal='oryx/127.0.97.253@KRBTEST.COM'} at 127.0.0.1:35010
I20260925 16:38:44.777311 7847 tablet_replica.cc:330] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: stopping tablet replica
I20260925 16:38:44.777510 7847 raft_consensus.cc:2229] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:44.777688 7847 raft_consensus.cc:2258] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:44.778486 7847 ts_tablet_manager.cc:1905] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:44.779497 7846 ts_tablet_manager.cc:1905] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:44.781147 7847 ts_tablet_manager.cc:1918] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:44.781231 7847 log.cc:1196] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/0345b30f7cf444a48d8b7431e044618a
I20260925 16:38:44.781535 7847 ts_tablet_manager.cc:1939] T 0345b30f7cf444a48d8b7431e044618a P 7fc9b331c63641e6be7aa44d07d8fe46: Deleting consensus metadata
I20260925 16:38:44.782397 7079 catalog_manager.cc:4841] TS 7fc9b331c63641e6be7aa44d07d8fe46 (127.0.97.195:32799): tablet 0345b30f7cf444a48d8b7431e044618a (table test-table [id=afd83e24d4d441c198a5e724461c0fe4]) successfully deleted
I20260925 16:38:44.782560 7846 ts_tablet_manager.cc:1918] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:44.782665 7846 log.cc:1196] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/0345b30f7cf444a48d8b7431e044618a
I20260925 16:38:44.782919 7846 ts_tablet_manager.cc:1939] T 0345b30f7cf444a48d8b7431e044618a P ed1e284ddd77479ca69c1288e0c15f42: Deleting consensus metadata
I20260925 16:38:44.783537 7076 catalog_manager.cc:4841] TS ed1e284ddd77479ca69c1288e0c15f42 (127.0.97.194:38797): tablet 0345b30f7cf444a48d8b7431e044618a (table test-table [id=afd83e24d4d441c198a5e724461c0fe4]) successfully deleted
I20260925 16:38:44.785975 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7462
I20260925 16:38:44.786109 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:44.785715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34965 (local address 127.0.0.1:48894)
0925 16:38:44.785948 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:44.786008 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260925 16:38:44.787516 7076 catalog_manager.cc:4584] TS 60ab6503654145068332e88f3ea709ac (127.0.97.193:34965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0345b30f7cf444a48d8b7431e044618a: Network error: Client connection negotiation failed: client connection to 127.0.97.193:34965: connect: Connection refused (error 111)
I20260925 16:38:44.792325 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7599
I20260925 16:38:44.798360 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 6955
I20260925 16:38:44.804103 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7064
2026-09-25T16:38:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5375 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:44 dist-test-slave-kxl8 krb5kdc[7850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:44 dist-test-slave-kxl8 krb5kdc[7850](info): set up 2 sockets
Sep 25 16:38:44 dist-test-slave-kxl8 krb5kdc[7850](info): commencing operation
krb5kdc: starting...
W20260925 16:38:45.853801 391 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:45 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab.
Entry for principal oryx/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab.
I20260925 16:38:45.972859 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44313
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab
--principal=oryx/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:46.080056 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:46.080349 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:46.080420 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:46.084463 7863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:46.084564 7863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:38:46.084597 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:46.084640 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:46.084697 7863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:46.084735 7863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:46.089978 7863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.254.keytab
--principal=oryx/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:46.091020 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:46.092042 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:46.100179 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.100354 7863 server_base.cc:1034] running on GCE node
W20260925 16:38:46.100291 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:46.100306 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.100978 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:46.102109 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:46.103281 7863 hybrid_clock.cc:648] HybridClock initialized: now 1790354326103258 us; error 34 us; skew 500 ppm
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:46.106096 7863 init.cc:377] Logged in from keytab as oryx/127.0.97.254@KRBTEST.COM (short username oryx)
I20260925 16:38:46.106947 7863 webserver.cc:460] Webserver started at http://127.0.97.254:39617/ using document root <none> and password file <none>
I20260925 16:38:46.107432 7863 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:46.107486 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:46.107638 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:46.109591 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "3a61b35a0e034a23b29a1d959bc358d4"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "4affa6f0efade5bff7b8e519e4e94e70"
server_key_iv: "89ca223228eee3cfe80d1bd2d9ad9a44"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.110077 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "3a61b35a0e034a23b29a1d959bc358d4"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "4affa6f0efade5bff7b8e519e4e94e70"
server_key_iv: "89ca223228eee3cfe80d1bd2d9ad9a44"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.113451 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:38:46.115644 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.116863 7863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:46.116990 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "3a61b35a0e034a23b29a1d959bc358d4"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "4affa6f0efade5bff7b8e519e4e94e70"
server_key_iv: "89ca223228eee3cfe80d1bd2d9ad9a44"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.117094 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:46.142299 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:46.146000 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:46.146206 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:46.154114 7863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44313
I20260925 16:38:46.154145 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44313 every 8 connection(s)
I20260925 16:38:46.155048 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:46.157760 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:46.158437 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7863
I20260925 16:38:46.158579 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:46.158896 391 external_mini_cluster.cc:1428] Setting key 60d58cdac587cf95dd92cf33cec3645a
I20260925 16:38:46.165035 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Bootstrap starting.
I20260925 16:38:46.167558 7941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:46.168283 7941 log.cc:824] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:46.170423 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: No bootstrap required, opened a new log
I20260925 16:38:46.174304 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } }
I20260925 16:38:46.174523 7941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:46.174582 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a61b35a0e034a23b29a1d959bc358d4, State: Initialized, Role: FOLLOWER
I20260925 16:38:46.175040 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.160507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.97.254:44313)
0925 16:38:46.161195 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.161216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.161271 (+ 55us) server_negotiation.cc:407] Connection header received
0925 16:38:46.162131 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.162154 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.162600 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.162981 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.163691 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.165137 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.166045 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.166756 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.166911 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.169519 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.169525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.169538 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.169562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.171775 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.172263 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.172267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.172271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.172402 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.172678 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.172680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.172681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.173392 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.173558 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.173874 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.174198 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":102,"threads_started":1}
I20260925 16:38:46.175045 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } }
I20260925 16:38:46.175222 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:46.175282 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:46.175371 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:46.176294 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } }
I20260925 16:38:46.176644 7941 leader_election.cc:302] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a61b35a0e034a23b29a1d959bc358d4; no voters:
I20260925 16:38:46.176970 7941 leader_election.cc:288] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:46.177078 7946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:46.177279 7946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a61b35a0e034a23b29a1d959bc358d4, State: Running, Role: LEADER
I20260925 16:38:46.177584 7946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } }
I20260925 16:38:46.178220 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:46.179335 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a61b35a0e034a23b29a1d959bc358d4. Latest consensus state: current_term: 1 leader_uuid: "3a61b35a0e034a23b29a1d959bc358d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } } }
I20260925 16:38:46.179468 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:46.179740 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a61b35a0e034a23b29a1d959bc358d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a61b35a0e034a23b29a1d959bc358d4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44313 } } }
I20260925 16:38:46.179870 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:46.183495 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:46.185627 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:46.189268 7955 catalog_manager.cc:1347] Generated new cluster ID: aeedfa1a17054af79dc69745d590304c
I20260925 16:38:46.189354 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:46.208968 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:46.209987 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:46.219579 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Generated new TSK 0
I20260925 16:38:46.220376 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
I20260925 16:38:46.251873 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44313
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:46.362258 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:46.362521 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:46.362567 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:46.366484 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:46.366539 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:46.366576 7967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:46.366609 7967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:46.371806 7967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:46.372768 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:46.373750 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:46.383222 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:46.383255 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.383442 7967 server_base.cc:1034] running on GCE node
W20260925 16:38:46.383437 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.383868 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:46.384397 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:46.385588 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790354326385576 us; error 25 us; skew 500 ppm
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:46.393788 7967 init.cc:377] Logged in from keytab as oryx/127.0.97.193@KRBTEST.COM (short username oryx)
I20260925 16:38:46.394683 7967 webserver.cc:460] Webserver started at http://127.0.97.193:43715/ using document root <none> and password file <none>
I20260925 16:38:46.395246 7967 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:46.395304 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:46.395515 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:46.397614 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "08ce6d188fde4443a15b46a858ee8ac6"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "be3b625bf6797e3102fd978af5b027a2"
server_key_iv: "11b6a32bc9caf2924c1089e2049b19fe"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.398162 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "08ce6d188fde4443a15b46a858ee8ac6"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "be3b625bf6797e3102fd978af5b027a2"
server_key_iv: "11b6a32bc9caf2924c1089e2049b19fe"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.401757 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:46.404048 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.405220 7967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:46.405397 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "08ce6d188fde4443a15b46a858ee8ac6"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "be3b625bf6797e3102fd978af5b027a2"
server_key_iv: "11b6a32bc9caf2924c1089e2049b19fe"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.405493 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:46.419785 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:46.422948 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:46.423144 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:46.424578 7967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:46.424650 7967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.424722 7967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:46.424773 7967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:46.435681 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:41423
I20260925 16:38:46.435710 8097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:41423 every 8 connection(s)
I20260925 16:38:46.436833 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:46.438309 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 7967
I20260925 16:38:46.438476 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:46.438795 391 external_mini_cluster.cc:1428] Setting key 94114871dc53541b28d7bda0df9a0d88
I20260925 16:38:46.439435 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.425399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:45993 (local address 127.0.97.254:44313)
0925 16:38:46.425534 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.425540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.426639 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 16:38:46.427670 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.427674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.427820 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.427888 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.429348 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.430050 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.431895 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.432291 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.432407 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.435264 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.435268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.435271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.435304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.436977 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.437566 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.437570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.437572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.437667 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.438121 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.438126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.438128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.438362 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.438467 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.439108 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.439245 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:38:46.440120 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.425941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.193:45993)
0925 16:38:46.426466 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.426523 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.427426 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.428301 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.428307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.428735 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.429178 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.429190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.430192 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.430199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.431756 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.431770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.432473 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.432480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.432688 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.433324 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.433348 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.435098 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.437080 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.437084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.437094 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.437459 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.437816 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.437819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.437822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.437982 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.438523 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.438532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.438841 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.439043 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.439336 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":140,"threads_started":1}
I20260925 16:38:46.441412 8098 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44313
I20260925 16:38:46.441558 8098 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:46.442128 8098 heartbeater.cc:511] Master 127.0.97.254:44313 requested a full tablet report, sending...
I20260925 16:38:46.443490 7894 ts_manager.cc:194] Registered new tserver with Master: 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423)
I20260925 16:38:46.444959 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:45993
WARNING: no policy specified for oryx/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:38:46.451417 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.443681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:33007 (local address 127.0.97.254:44313)
0925 16:38:46.443785 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.443789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.443942 (+ 153us) server_negotiation.cc:407] Connection header received
0925 16:38:46.444030 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.444036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.444089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.444171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.444651 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.445530 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.446887 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.447138 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.447219 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.448447 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.448450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.448452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.448472 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.449984 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.450348 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.450350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.450351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.450419 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.450666 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.450670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.450672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.450903 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.450985 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.451217 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.451305 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":30}
I20260925 16:38:46.451457 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.443559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.193:33007)
0925 16:38:46.443847 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.443862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.443957 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.444204 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.444207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.444419 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.444533 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.444540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.445684 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.445688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.446737 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.446751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.447256 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.447262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.447359 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.447825 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.447844 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.448315 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.450081 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.450084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.450085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.450261 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.450495 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.450497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.450498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.450566 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.451011 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.451014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.451088 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.451188 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.451301 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
I20260925 16:38:46.468748 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44313
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:46.580557 8107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:46.580829 8107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:46.580904 8107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:46.584897 8107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:46.584985 8107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:46.585058 8107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:46.585139 8107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:46.590339 8107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:46.591356 8107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:46.592342 8107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:46.601722 8115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:46.601780 8112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.601754 8107 server_base.cc:1034] running on GCE node
W20260925 16:38:46.601737 8113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.602442 8107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:46.602995 8107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:46.604153 8107 hybrid_clock.cc:648] HybridClock initialized: now 1790354326604132 us; error 38 us; skew 500 ppm
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:46.606819 8107 init.cc:377] Logged in from keytab as oryx/127.0.97.194@KRBTEST.COM (short username oryx)
I20260925 16:38:46.607667 8107 webserver.cc:460] Webserver started at http://127.0.97.194:38097/ using document root <none> and password file <none>
I20260925 16:38:46.608163 8107 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:46.608219 8107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:46.608385 8107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:46.610380 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "1f977546aa724138ae3c29a7fdd347a1"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "9b040c1fb689ac86154f6c68e1ba083e"
server_key_iv: "40ada97b7207f9c9512106dcbafa13af"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.610869 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "1f977546aa724138ae3c29a7fdd347a1"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "9b040c1fb689ac86154f6c68e1ba083e"
server_key_iv: "40ada97b7207f9c9512106dcbafa13af"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.614296 8107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:46.616483 8122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.617791 8107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:46.617923 8107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "1f977546aa724138ae3c29a7fdd347a1"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "9b040c1fb689ac86154f6c68e1ba083e"
server_key_iv: "40ada97b7207f9c9512106dcbafa13af"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.618026 8107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:46.628862 8107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:46.631440 8107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:46.631620 8107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:46.632922 8107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:46.632999 8107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.633064 8107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:46.633100 8107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:46.643953 8107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:43605
I20260925 16:38:46.644075 8237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:43605 every 8 connection(s)
I20260925 16:38:46.645022 8107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:46.646806 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.633597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:35951 (local address 127.0.97.254:44313)
0925 16:38:46.633723 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.633728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.634737 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 16:38:46.635638 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.635642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.635698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.635782 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.637127 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.637851 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.639127 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.639542 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.639661 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.643051 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.643055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.643058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.643084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.644782 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.645365 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.645371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.645373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.645470 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.645844 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.645848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.645850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.646076 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.646173 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.646593 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.646689 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:38:46.647545 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.634160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.194:35951)
0925 16:38:46.634601 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.634640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.635442 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.636056 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.636070 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.636371 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.636989 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.637004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.637973 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.637977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.638988 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.638997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.639749 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.639755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.639960 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.640545 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.640560 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.642882 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.644887 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.644891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.644900 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.645209 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.645561 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.645566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.645568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.645731 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.646179 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.646184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.646390 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.646539 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.646763 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260925 16:38:46.648706 8238 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44313
I20260925 16:38:46.648836 8238 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:46.649376 8238 heartbeater.cc:511] Master 127.0.97.254:44313 requested a full tablet report, sending...
I20260925 16:38:46.650388 7895 ts_manager.cc:194] Registered new tserver with Master: 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605)
I20260925 16:38:46.651161 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:35951
I20260925 16:38:46.654733 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8107
I20260925 16:38:46.654878 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:46.655175 391 external_mini_cluster.cc:1428] Setting key b12e26359ca386ac3f654642cb902214
I20260925 16:38:46.658856 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.650978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:36025 (local address 127.0.97.254:44313)
0925 16:38:46.651096 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.651101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.651253 (+ 152us) server_negotiation.cc:407] Connection header received
0925 16:38:46.651394 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.651397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.651436 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.651525 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.652007 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.652482 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.653605 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.653872 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.653961 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.655201 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.655208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.655213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.655239 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.656952 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.657418 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.657425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.657429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.657520 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.657788 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.657793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.657798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.658090 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.658196 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.658589 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.658706 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:38:46.658877 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.650901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.194:36025)
0925 16:38:46.651175 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.651186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.651293 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.651538 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.651541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.651768 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.651892 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.651900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.652608 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.652611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.653450 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.653460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.653975 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.653978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.654043 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.654598 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.654609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.655068 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.657061 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.657064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.657065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.657327 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.657614 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.657616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.657618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.657697 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.658270 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.658273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.658404 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.658531 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.658700 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
I20260925 16:38:46.682308 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44313
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:46.792102 8247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:46.792362 8247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:46.792464 8247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:46.796558 8247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:46.796619 8247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:46.796659 8247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:38:46.796757 8247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:46.801967 8247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.254:44313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:46.803017 8247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:46.804044 8247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:46.813623 8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:46.813623 8253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:46.813597 8252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:46.813979 8247 server_base.cc:1034] running on GCE node
I20260925 16:38:46.814383 8247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:46.815003 8247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:46.816200 8247 hybrid_clock.cc:648] HybridClock initialized: now 1790354326816177 us; error 33 us; skew 500 ppm
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:46.819007 8247 init.cc:377] Logged in from keytab as oryx/127.0.97.195@KRBTEST.COM (short username oryx)
I20260925 16:38:46.819948 8247 webserver.cc:460] Webserver started at http://127.0.97.195:33965/ using document root <none> and password file <none>
I20260925 16:38:46.820514 8247 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:46.820600 8247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:46.820845 8247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:46.823065 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "12d52b977302476298d5bf38bb3e6ff0"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "99c187adcb0bea5fb6bbd16653fbc6e8"
server_key_iv: "91a7cc1fc0668aca352bfd4074cd1758"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.823654 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "12d52b977302476298d5bf38bb3e6ff0"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "99c187adcb0bea5fb6bbd16653fbc6e8"
server_key_iv: "91a7cc1fc0668aca352bfd4074cd1758"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.827430 8247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:38:46.829929 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.831159 8247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:46.831301 8247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "12d52b977302476298d5bf38bb3e6ff0"
format_stamp: "Formatted at 2026-09-25 16:38:46 on dist-test-slave-kxl8"
server_key: "99c187adcb0bea5fb6bbd16653fbc6e8"
server_key_iv: "91a7cc1fc0668aca352bfd4074cd1758"
server_key_version: "encryptionkey@0"
I20260925 16:38:46.831427 8247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:46.842839 8247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:46.845610 8247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:46.845806 8247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:46.847198 8247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:46.847270 8247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:46.847339 8247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:46.847365 8247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:46.858242 8247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:42721
I20260925 16:38:46.858311 8377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:42721 every 8 connection(s)
I20260925 16:38:46.859164 8247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:46.860999 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.847872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:50593 (local address 127.0.97.254:44313)
0925 16:38:46.847998 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.848001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.849010 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 16:38:46.850050 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.850054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.850107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.850191 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.851536 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.852444 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.853824 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.854233 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.854367 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.857396 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.857400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.857402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.857424 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.858967 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.859551 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.859555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.859557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.859648 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.860021 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.860025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.860026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.860241 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.860318 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.860787 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.860889 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:38:46.861727 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.848369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.195:50593)
0925 16:38:46.848864 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.848917 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.849809 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.850474 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.850485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.850846 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.851363 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.851376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.852622 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.852626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.853718 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.853728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.854434 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.854441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.854649 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.855251 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.855270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.857163 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.859080 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.859088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.859103 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.859424 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.859755 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.859758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.859759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.859921 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.860344 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.860348 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.860544 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.860682 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.860904 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260925 16:38:46.862867 8378 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44313
I20260925 16:38:46.863062 8378 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:46.863565 8378 heartbeater.cc:511] Master 127.0.97.254:44313 requested a full tablet report, sending...
I20260925 16:38:46.864619 7894 ts_manager.cc:194] Registered new tserver with Master: 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:46.865414 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:50593
I20260925 16:38:46.868091 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8247
I20260925 16:38:46.868259 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:46.868589 391 external_mini_cluster.cc:1428] Setting key b3ebad87e121c0759c91fb4c79d1ecc2
I20260925 16:38:46.870728 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:38:46.872051 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.865206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:43947 (local address 127.0.97.254:44313)
0925 16:38:46.865367 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.865371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.865535 (+ 164us) server_negotiation.cc:407] Connection header received
0925 16:38:46.865650 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.865653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.865698 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.865785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.866209 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.866703 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.867516 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.867732 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.867860 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.869024 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.869027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.869030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.869053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.870464 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.870896 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.870901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.870903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.870978 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.871248 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.871253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.871255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.871422 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.871527 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.871784 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.871884 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:38:46.872109 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.865139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.97.195:43947)
0925 16:38:46.865455 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.865468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.865575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.865804 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.865807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.865997 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:46.866101 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.866107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.866836 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.866839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.867392 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.867406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.867850 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.867853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.867944 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.868496 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:46.868507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:46.868895 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:46.870586 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.870589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.870591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.870804 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.871070 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:46.871075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:46.871077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:46.871160 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:46.871527 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:46.871531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:46.871623 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.871730 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.871878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:46.894532 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:46.888339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.97.254:44313)
0925 16:38:46.888470 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.888474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.888487 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:46.888525 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.888527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.888564 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.888634 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.889414 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.889938 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.890801 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.891012 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.891119 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.894386 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35384: BlockingRecv error: recv got EOF from 127.0.0.1:35384 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 16:38:46 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:46.904070 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.895983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.254:44313)
0925 16:38:46.896105 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.896109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.896278 (+ 169us) server_negotiation.cc:407] Connection header received
0925 16:38:46.896355 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.896357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.896392 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.896457 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:46.897068 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.897628 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.898494 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.898689 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.898761 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.901016 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:46.901019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:46.901021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:46.901042 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:46.902472 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.902894 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.902899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.902901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.902980 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:46.903249 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:46.903254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:46.903255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:46.903433 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:46.903580 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.903804 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.903906 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:38:46.906874 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:46.910036 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:46.924222 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.918157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:43605 (local address 127.0.0.1:39988)
0925 16:38:46.918523 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.918573 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.918702 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.919660 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.919668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.919691 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:46.919856 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.919863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921309 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.921315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.922684 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.922694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.923664 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.923668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.923768 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.923803 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.923950 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.924087 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260925 16:38:46.924233 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.918301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.97.194:43605)
0925 16:38:46.918995 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.919007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.919032 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:38:46.919111 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.919116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.919494 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.920010 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:46.920150 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921184 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.922853 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.923543 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.923665 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.923844 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.923978 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.924079 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":96,"threads_started":1}
I20260925 16:38:46.924549 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.918536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:41423 (local address 127.0.0.1:44078)
0925 16:38:46.918913 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.918931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.919015 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.920123 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.920129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.920148 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:46.920302 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.920309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921812 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.921814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.922942 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.922950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.924147 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.924150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.924230 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.924248 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.924324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.924411 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260925 16:38:46.924566 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.918864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:42721 (local address 127.0.0.1:43660)
0925 16:38:46.919297 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.919312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.919392 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.920148 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.920150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.920162 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:46.920345 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.920353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921764 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.921769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.923112 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.923120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.924099 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.924103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.924204 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.924229 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.924277 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.924471 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":259,"thread_start_us":92,"threads_started":1}
I20260925 16:38:46.925002 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.918793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.97.193:41423)
0925 16:38:46.919622 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.919636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.919655 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:38:46.919720 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.919725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.919954 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.920103 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:46.920435 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921688 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.923061 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.923829 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.923935 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.924100 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.924745 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.924852 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":75,"threads_started":1}
I20260925 16:38:46.925581 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.919277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.97.195:42721)
0925 16:38:46.919680 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.919692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.919713 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:38:46.919785 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.919790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.919976 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.920286 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:46.920451 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.921638 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.923223 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.923973 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.924974 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.925133 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.925180 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.925265 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260925 16:38:46.927433 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 79860d1a3325445195f6a9c33da0fd7c (DEFAULT_TABLE table=test-table [id=8d15c63761ba4e92a02061904d3f104a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:46.927433 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 79860d1a3325445195f6a9c33da0fd7c (DEFAULT_TABLE table=test-table [id=8d15c63761ba4e92a02061904d3f104a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:46.927690 8293 tablet_service.cc:1467] Processing CreateTablet for tablet 79860d1a3325445195f6a9c33da0fd7c (DEFAULT_TABLE table=test-table [id=8d15c63761ba4e92a02061904d3f104a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:46.928524 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79860d1a3325445195f6a9c33da0fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:46.928524 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79860d1a3325445195f6a9c33da0fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:46.928552 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79860d1a3325445195f6a9c33da0fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:46.935084 8406 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Bootstrap starting.
I20260925 16:38:46.936712 8408 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Bootstrap starting.
I20260925 16:38:46.937410 8406 tablet_bootstrap.cc:654] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:46.937600 8407 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Bootstrap starting.
I20260925 16:38:46.938277 8406 log.cc:824] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:46.938624 8408 tablet_bootstrap.cc:654] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:46.939455 8408 log.cc:824] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:46.939492 8407 tablet_bootstrap.cc:654] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:46.940388 8407 log.cc:824] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:46.940464 8406 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: No bootstrap required, opened a new log
I20260925 16:38:46.940630 8406 ts_tablet_manager.cc:1397] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260925 16:38:46.942348 8407 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: No bootstrap required, opened a new log
I20260925 16:38:46.942549 8407 ts_tablet_manager.cc:1397] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:38:46.944634 8406 raft_consensus.cc:348] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.944911 8406 raft_consensus.cc:374] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:46.944970 8406 raft_consensus.cc:729] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d52b977302476298d5bf38bb3e6ff0, State: Initialized, Role: FOLLOWER
I20260925 16:38:46.945451 8406 consensus_queue.cc:260] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.946137 8378 heartbeater.cc:503] Master 127.0.97.254:44313 was elected leader, sending a full tablet report...
I20260925 16:38:46.946553 8406 ts_tablet_manager.cc:1428] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:38:46.947240 8407 raft_consensus.cc:348] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.947463 8408 tablet_bootstrap.cc:492] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: No bootstrap required, opened a new log
I20260925 16:38:46.947518 8407 raft_consensus.cc:374] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:46.947580 8407 raft_consensus.cc:729] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:46.947651 8408 ts_tablet_manager.cc:1397] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260925 16:38:46.948117 8407 consensus_queue.cc:260] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.948781 8238 heartbeater.cc:503] Master 127.0.97.254:44313 was elected leader, sending a full tablet report...
I20260925 16:38:46.949107 8407 ts_tablet_manager.cc:1428] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:38:46.952338 8408 raft_consensus.cc:348] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.952636 8408 raft_consensus.cc:374] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:46.952708 8408 raft_consensus.cc:729] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Initialized, Role: FOLLOWER
I20260925 16:38:46.953171 8408 consensus_queue.cc:260] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.953918 8098 heartbeater.cc:503] Master 127.0.97.254:44313 was elected leader, sending a full tablet report...
I20260925 16:38:46.954272 8408 ts_tablet_manager.cc:1428] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260925 16:38:46.968400 8414 raft_consensus.cc:482] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:46.968636 8414 raft_consensus.cc:504] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.969873 8414 leader_election.cc:288] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:46.975082 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.970624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:58119 (local address 127.0.97.195:42721)
0925 16:38:46.970763 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.970769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.970862 (+ 93us) server_negotiation.cc:407] Connection header received
0925 16:38:46.971009 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.971012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.971063 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.971149 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:46.971385 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.972088 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.973724 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.974348 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.974456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.974507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.974886 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.974955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:38:46.975082 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.970217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34151 (local address 127.0.97.194:43605)
0925 16:38:46.970386 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.970392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.970511 (+ 119us) server_negotiation.cc:407] Connection header received
0925 16:38:46.970710 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.970714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.970774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.970816 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:46.971165 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.971981 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.973724 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.974333 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.974456 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.974507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:46.974886 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 16:38:46.974955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:38:46.975160 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.970497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:42721 (local address 127.0.97.193:58119)
0925 16:38:46.970756 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.970782 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.970905 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.971185 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.971187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.971200 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:46.971284 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.971288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.972181 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.972184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.973586 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.973597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.974482 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.974486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.974714 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.974768 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.974847 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.974979 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":196,"thread_start_us":76,"threads_started":1}
I20260925 16:38:46.975382 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.970056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:43605 (local address 127.0.97.193:34151)
0925 16:38:46.970409 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:46.970426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:46.970564 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:46.970936 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:46.970938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:46.970953 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:46.971054 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.971058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.972094 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.972097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.973586 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:46.973597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.974481 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:46.974486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:46.974700 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:46.974764 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 16:38:46.974847 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:38:46.974980 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":218,"thread_start_us":105,"threads_started":1}
I20260925 16:38:46.975787 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79860d1a3325445195f6a9c33da0fd7c" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0" is_pre_election: true
I20260925 16:38:46.976027 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79860d1a3325445195f6a9c33da0fd7c" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f977546aa724138ae3c29a7fdd347a1" is_pre_election: true
I20260925 16:38:46.976082 8332 raft_consensus.cc:2454] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 0.
I20260925 16:38:46.976322 8191 raft_consensus.cc:2454] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 0.
I20260925 16:38:46.976568 7985 leader_election.cc:302] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 12d52b977302476298d5bf38bb3e6ff0; no voters:
I20260925 16:38:46.977061 8414 raft_consensus.cc:2781] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:46.977136 8414 raft_consensus.cc:482] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:46.977165 8414 raft_consensus.cc:3037] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:46.978379 8414 raft_consensus.cc:504] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.978735 8414 leader_election.cc:288] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:46.979038 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79860d1a3325445195f6a9c33da0fd7c" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f977546aa724138ae3c29a7fdd347a1"
I20260925 16:38:46.979187 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79860d1a3325445195f6a9c33da0fd7c" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0"
I20260925 16:38:46.979200 8191 raft_consensus.cc:3037] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:46.979305 8332 raft_consensus.cc:3037] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:46.980597 8332 raft_consensus.cc:2454] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 1.
I20260925 16:38:46.980890 7985 leader_election.cc:302] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 12d52b977302476298d5bf38bb3e6ff0; no voters:
I20260925 16:38:46.981040 8191 raft_consensus.cc:2454] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 1.
I20260925 16:38:46.981086 8414 raft_consensus.cc:2781] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:46.981315 8414 raft_consensus.cc:686] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 LEADER]: Becoming Leader. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Running, Role: LEADER
I20260925 16:38:46.981742 8414 consensus_queue.cc:237] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } }
I20260925 16:38:46.984987 7895 catalog_manager.cc:5495] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "08ce6d188fde4443a15b46a858ee8ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } health_report { overall_health: HEALTHY } } }
I20260925 16:38:47.001899 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:46.998707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.0.97.193:41423)
0925 16:38:46.998829 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:46.998833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:46.998866 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:38:46.998975 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:46.998977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:46.999031 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:46.999111 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:46.999306 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:46.999799 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.000582 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.000783 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.000856 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.001079 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.001592 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.001678 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.001740 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.001789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:47.007561 8191 raft_consensus.cc:1261] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Refusing update from remote peer 08ce6d188fde4443a15b46a858ee8ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.007561 8332 raft_consensus.cc:1261] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Refusing update from remote peer 08ce6d188fde4443a15b46a858ee8ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.008263 8414 consensus_queue.cc:1035] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.008494 8417 consensus_queue.cc:1035] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.016811 8421 mvcc.cc:205] Tried to move back new op lower bound from 7333291323415814144 to 7333291323320315904. Current Snapshot: MvccSnapshot[applied={T|T < 7333291323415814144}]
I20260925 16:38:47.017659 8422 mvcc.cc:205] Tried to move back new op lower bound from 7333291323415814144 to 7333291323320315904. Current Snapshot: MvccSnapshot[applied={T|T < 7333291323415814144}]
I20260925 16:38:47.025593 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.025736 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.027860 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Sending DeleteTablet for 3 replicas of tablet 79860d1a3325445195f6a9c33da0fd7c
I20260925 16:38:47.028666 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 79860d1a3325445195f6a9c33da0fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 16:38:47.028712 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet 79860d1a3325445195f6a9c33da0fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:43660
I20260925 16:38:47.028851 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet 79860d1a3325445195f6a9c33da0fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:39988
I20260925 16:38:47.029228 8431 tablet_replica.cc:330] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: stopping tablet replica
I20260925 16:38:47.029455 8432 tablet_replica.cc:330] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: stopping tablet replica
I20260925 16:38:47.029543 8431 raft_consensus.cc:2229] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:47.029788 8432 raft_consensus.cc:2229] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.029942 8431 raft_consensus.cc:2258] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.029950 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:47.030083 8432 raft_consensus.cc:2258] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:38:47.030349 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:47.030418 8430 tablet_replica.cc:330] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: stopping tablet replica
I20260925 16:38:47.030679 8430 raft_consensus.cc:2229] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.030942 8430 raft_consensus.cc:2258] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.031368 8431 ts_tablet_manager.cc:1905] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.032379 8430 ts_tablet_manager.cc:1905] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.034166 8431 ts_tablet_manager.cc:1918] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.034263 8431 log.cc:1196] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/79860d1a3325445195f6a9c33da0fd7c
I20260925 16:38:47.034596 8431 ts_tablet_manager.cc:1939] T 79860d1a3325445195f6a9c33da0fd7c P 08ce6d188fde4443a15b46a858ee8ac6: Deleting consensus metadata
I20260925 16:38:47.035146 8430 ts_tablet_manager.cc:1918] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.035228 8430 log.cc:1196] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/79860d1a3325445195f6a9c33da0fd7c
I20260925 16:38:47.035530 8430 ts_tablet_manager.cc:1939] T 79860d1a3325445195f6a9c33da0fd7c P 12d52b977302476298d5bf38bb3e6ff0: Deleting consensus metadata
I20260925 16:38:47.035842 7882 catalog_manager.cc:4841] TS 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423): tablet 79860d1a3325445195f6a9c33da0fd7c (table test-table [id=8d15c63761ba4e92a02061904d3f104a]) successfully deleted
I20260925 16:38:47.036729 7881 catalog_manager.cc:4841] TS 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721): tablet 79860d1a3325445195f6a9c33da0fd7c (table test-table [id=8d15c63761ba4e92a02061904d3f104a]) successfully deleted
I20260925 16:38:47.036712 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f (DEFAULT_TABLE table=test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.036967 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944188667ae41f4ba3bdc7d86f14c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.037199 8293 tablet_service.cc:1467] Processing CreateTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f (DEFAULT_TABLE table=test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.037453 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944188667ae41f4ba3bdc7d86f14c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.037408 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f (DEFAULT_TABLE table=test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.037647 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944188667ae41f4ba3bdc7d86f14c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.039091 8408 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Bootstrap starting.
I20260925 16:38:47.039506 8406 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Bootstrap starting.
I20260925 16:38:47.040099 8407 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Bootstrap starting.
I20260925 16:38:47.040115 8408 tablet_bootstrap.cc:654] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.040444 8406 tablet_bootstrap.cc:654] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.041083 8407 tablet_bootstrap.cc:654] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.041679 8406 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: No bootstrap required, opened a new log
I20260925 16:38:47.041764 8406 ts_tablet_manager.cc:1397] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 16:38:47.042225 8432 ts_tablet_manager.cc:1905] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.042303 8406 raft_consensus.cc:348] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.042411 8406 raft_consensus.cc:374] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.042451 8406 raft_consensus.cc:729] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d52b977302476298d5bf38bb3e6ff0, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.042586 8406 consensus_queue.cc:260] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.042865 8406 ts_tablet_manager.cc:1428] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:38:47.045740 8432 ts_tablet_manager.cc:1918] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.045822 8432 log.cc:1196] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/79860d1a3325445195f6a9c33da0fd7c
I20260925 16:38:47.046113 8432 ts_tablet_manager.cc:1939] T 79860d1a3325445195f6a9c33da0fd7c P 1f977546aa724138ae3c29a7fdd347a1: Deleting consensus metadata
I20260925 16:38:47.047206 7881 catalog_manager.cc:4841] TS 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605): tablet 79860d1a3325445195f6a9c33da0fd7c (table test-table [id=8d15c63761ba4e92a02061904d3f104a]) successfully deleted
I20260925 16:38:47.050135 8408 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: No bootstrap required, opened a new log
I20260925 16:38:47.050249 8408 ts_tablet_manager.cc:1397] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260925 16:38:47.050727 8408 raft_consensus.cc:348] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.050851 8408 raft_consensus.cc:374] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.050884 8408 raft_consensus.cc:729] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.051038 8408 consensus_queue.cc:260] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.051306 8408 ts_tablet_manager.cc:1428] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 16:38:47.051887 8407 tablet_bootstrap.cc:492] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: No bootstrap required, opened a new log
I20260925 16:38:47.051981 8407 ts_tablet_manager.cc:1397] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 16:38:47.052416 8407 raft_consensus.cc:348] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.052533 8407 raft_consensus.cc:374] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.052565 8407 raft_consensus.cc:729] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.052697 8407 consensus_queue.cc:260] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.052964 8407 ts_tablet_manager.cc:1428] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:47.101176 8413 raft_consensus.cc:482] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:47.101356 8413 raft_consensus.cc:504] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.102627 8413 leader_election.cc:288] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.106536 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.102911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:42721 (local address 127.0.97.194:58425)
0925 16:38:47.103102 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:47.103117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:47.103212 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:47.103480 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:47.103483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:47.103503 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:47.103645 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.103652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.104608 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.104615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.105490 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.105499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.106117 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.106121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.106226 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.106243 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:38:47.106338 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 16:38:47.106409 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 16:38:47.106536 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.103094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:58425 (local address 127.0.97.195:42721)
0925 16:38:47.103214 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.103219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.103247 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:38:47.103300 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.103303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.103350 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.103440 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:47.103832 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.104516 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.105609 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.105997 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.106128 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.106180 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.106352 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.106416 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:38:47.106992 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1944188667ae41f4ba3bdc7d86f14c1f" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0" is_pre_election: true
I20260925 16:38:47.107276 8332 raft_consensus.cc:2454] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 0.
I20260925 16:38:47.107751 8125 leader_election.cc:302] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d52b977302476298d5bf38bb3e6ff0, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.107910 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.102982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:59183 (local address 127.0.97.193:41423)
0925 16:38:47.103098 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.103103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.103610 (+ 507us) server_negotiation.cc:407] Connection header received
0925 16:38:47.103669 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.103672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.103716 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.103787 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:38:47.104145 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.105025 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.106609 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.107211 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.107298 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.107347 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.107721 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.107794 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:38:47.108065 8413 raft_consensus.cc:2781] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:47.108121 8413 raft_consensus.cc:482] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:47.108158 8413 raft_consensus.cc:3037] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.108263 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.102911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:41423 (local address 127.0.97.194:59183)
0925 16:38:47.103487 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:47.103500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:47.103592 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:47.103888 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:47.103892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:47.103908 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:38:47.104024 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.104030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.105181 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.105184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.106456 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.106467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.107452 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.107458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.107580 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.107601 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:38:47.107681 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:38:47.107749 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":235,"thread_start_us":87,"threads_started":1}
I20260925 16:38:47.108981 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1944188667ae41f4ba3bdc7d86f14c1f" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6" is_pre_election: true
I20260925 16:38:47.109227 8052 raft_consensus.cc:2454] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 0.
I20260925 16:38:47.109683 8413 raft_consensus.cc:504] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
W20260925 16:38:47.110066 8379 tablet.cc:2369] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:47.110074 8413 leader_election.cc:288] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Requested vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.110452 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1944188667ae41f4ba3bdc7d86f14c1f" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0"
I20260925 16:38:47.110580 8332 raft_consensus.cc:3037] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.110554 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1944188667ae41f4ba3bdc7d86f14c1f" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6"
I20260925 16:38:47.110659 8052 raft_consensus.cc:3037] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.111711 8332 raft_consensus.cc:2454] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.111711 8052 raft_consensus.cc:2454] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.111990 8126 leader_election.cc:302] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.112176 8413 raft_consensus.cc:2781] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:47.112457 8413 raft_consensus.cc:686] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Running, Role: LEADER
I20260925 16:38:47.112725 8413 consensus_queue.cc:237] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.115099 7892 catalog_manager.cc:5495] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.254@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
W20260925 16:38:47.146303 8239 tablet.cc:2369] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:47.148605 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.139081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44313 (local address 127.0.0.1:35412)
0925 16:38:47.139416 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:47.139427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:47.139573 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:47.139830 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:47.139832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:47.140314 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:47.140405 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.140412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.141292 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.141298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.142296 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.142308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.142767 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.142771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.142859 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.143504 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:47.143519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:47.145081 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:47.146986 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:47.146998 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:47.147005 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:47.147216 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:47.147492 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:47.147497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:47.147499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:47.147575 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:47.147959 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:47.147968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:47.148066 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 16:38:47.148180 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 16:38:47.148403 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260925 16:38:47.148604 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.139148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.0.97.254:44313)
0925 16:38:47.139275 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.139279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.139511 (+ 232us) server_negotiation.cc:407] Connection header received
0925 16:38:47.139678 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.139681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.139723 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.139807 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:47.140523 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.141078 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.142420 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.142627 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.142709 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.145229 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:47.145233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:47.145236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:47.145267 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:47.146878 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:47.147316 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:47.147320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:47.147322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:47.147399 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:47.147663 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:47.147669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:47.147671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:47.147846 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:47.147950 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.148258 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.148477 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":48}
I20260925 16:38:47.153234 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:35412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:38:47.154089 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:47.159300 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4a2308a05b4c7aa3166a95ea69680e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b1fba3475254d84b7991e51e038e331]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:47.159716 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4a2308a05b4c7aa3166a95ea69680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.159821 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4a2308a05b4c7aa3166a95ea69680e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b1fba3475254d84b7991e51e038e331]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:47.160208 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4a2308a05b4c7aa3166a95ea69680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.162684 8408 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: Bootstrap starting.
I20260925 16:38:47.163937 8408 tablet_bootstrap.cc:654] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.164789 8293 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4a2308a05b4c7aa3166a95ea69680e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b1fba3475254d84b7991e51e038e331]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:38:47.165145 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4a2308a05b4c7aa3166a95ea69680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.165767 8408 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: No bootstrap required, opened a new log
I20260925 16:38:47.165848 8408 ts_tablet_manager.cc:1397] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 16:38:47.166556 8408 raft_consensus.cc:348] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.166666 8408 raft_consensus.cc:374] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.166694 8408 raft_consensus.cc:729] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.166814 8408 consensus_queue.cc:260] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.167073 8408 ts_tablet_manager.cc:1428] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:47.168380 8417 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.168524 8417 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.168761 8407 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: Bootstrap starting.
I20260925 16:38:47.169735 8407 tablet_bootstrap.cc:654] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.172627 8407 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: No bootstrap required, opened a new log
I20260925 16:38:47.172731 8407 ts_tablet_manager.cc:1397] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:38:47.173470 8407 raft_consensus.cc:348] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.173588 8407 raft_consensus.cc:374] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.173632 8407 raft_consensus.cc:729] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.173774 8407 consensus_queue.cc:260] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.174139 8413 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.174276 8413 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.174363 8406 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: Bootstrap starting.
I20260925 16:38:47.175374 8406 tablet_bootstrap.cc:654] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.175366 8413 raft_consensus.cc:482] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:47.175442 8413 raft_consensus.cc:504] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.175812 8413 leader_election.cc:288] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.176110 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4a2308a05b4c7aa3166a95ea69680e" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6" is_pre_election: true
I20260925 16:38:47.176232 8052 raft_consensus.cc:2454] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 0.
I20260925 16:38:47.176299 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4a2308a05b4c7aa3166a95ea69680e" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0" is_pre_election: true
I20260925 16:38:47.176527 8126 leader_election.cc:302] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.176753 8413 raft_consensus.cc:2781] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:47.176811 8413 raft_consensus.cc:482] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:47.176853 8413 raft_consensus.cc:3037] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Advancing to term 1
W20260925 16:38:47.176940 8125 leader_election.cc:341] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721): Illegal state: must be running to vote when last-logged opid is not known
I20260925 16:38:47.174192 8407 ts_tablet_manager.cc:1428] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:47.177465 8406 tablet_bootstrap.cc:492] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: No bootstrap required, opened a new log
I20260925 16:38:47.177546 8406 ts_tablet_manager.cc:1397] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:38:47.177974 8413 raft_consensus.cc:504] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.177944 8406 raft_consensus.cc:348] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.178051 8406 raft_consensus.cc:374] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.178079 8406 raft_consensus.cc:729] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d52b977302476298d5bf38bb3e6ff0, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.178198 8406 consensus_queue.cc:260] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.178313 8413 leader_election.cc:288] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Requested vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.178432 8406 ts_tablet_manager.cc:1428] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:47.178654 8412 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.178695 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4a2308a05b4c7aa3166a95ea69680e" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6"
I20260925 16:38:47.178800 8412 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.178844 8052 raft_consensus.cc:3037] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.179226 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4a2308a05b4c7aa3166a95ea69680e" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0"
I20260925 16:38:47.179317 8332 raft_consensus.cc:3037] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.179976 8052 raft_consensus.cc:2454] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.180258 8332 raft_consensus.cc:2454] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.180320 8126 leader_election.cc:302] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.180531 8413 raft_consensus.cc:2781] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:47.180630 8413 raft_consensus.cc:686] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Running, Role: LEADER
I20260925 16:38:47.180743 8413 consensus_queue.cc:237] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.181488 8437 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: TxnStatusTablet state changed. Reason: New leader 1f977546aa724138ae3c29a7fdd347a1. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.181581 8437 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:47.181811 8449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:38:47.181968 8449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:38:47.182966 7892 catalog_manager.cc:5495] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } health_report { overall_health: UNKNOWN } } }
W20260925 16:38:47.188107 8099 tablet.cc:2369] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:38:47.199266 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.193968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:43605 (local address 127.0.0.1:39992)
0925 16:38:47.194119 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:47.194137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:47.194249 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:47.194666 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:47.194669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:47.194677 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:47.194775 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.194781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.195917 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.195920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.196943 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.196953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.197717 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.197721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.197828 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.197879 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.198931 (+ 1052us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.198945 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 16:38:47.199013 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:38:47.199099 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 16:38:47.199721 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.194088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.0.97.194:43605)
0925 16:38:47.194241 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.194245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.194260 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:47.194393 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.194396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.194462 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.194549 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:47.194950 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.195751 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.197077 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.197519 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.197627 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.198051 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.198583 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.198684 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.199525 (+ 841us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.199582 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:38:47.202522 8332 raft_consensus.cc:1261] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.202652 8052 raft_consensus.cc:1261] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.203127 8437 consensus_queue.cc:1035] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.203512 8413 consensus_queue.cc:1035] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.204833 8412 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: TxnStatusTablet state changed. Reason: New leader 1f977546aa724138ae3c29a7fdd347a1. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.204950 8412 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.204969 8417 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: TxnStatusTablet state changed. Reason: New leader 1f977546aa724138ae3c29a7fdd347a1. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.205082 8417 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.206574 8437 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.206672 8437 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:47.206769 8437 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.206826 8437 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 1f977546aa724138ae3c29a7fdd347a1: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:38:47.208058 8412 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.208158 8412 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 12d52b977302476298d5bf38bb3e6ff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.208527 8417 tablet_replica.cc:439] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } } }
I20260925 16:38:47.208626 8417 tablet_replica.cc:442] T 6e4a2308a05b4c7aa3166a95ea69680e P 08ce6d188fde4443a15b46a858ee8ac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:38:47.215420 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.212153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.0.97.194:43605)
0925 16:38:47.212304 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.212308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.212320 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:47.212358 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.212361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.212403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.212482 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:47.212723 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.213525 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.214309 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.214656 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.214751 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.215032 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.215105 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.215176 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.215251 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.215296 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:38:47.225844 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.222097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:46125 (local address 127.0.97.194:43605)
0925 16:38:47.222217 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.222221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.222459 (+ 238us) server_negotiation.cc:407] Connection header received
0925 16:38:47.222559 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.222562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.222609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.222684 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:47.222946 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.223707 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.224645 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.224978 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.225077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.225399 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.225469 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.225538 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.225645 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.225687 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:38:47.225850 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.222027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:43605 (local address 127.0.97.194:46125)
0925 16:38:47.222342 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:47.222355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:47.222484 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:47.222696 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:47.222698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:47.222703 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:38:47.222834 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.222840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.223821 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.223823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.224508 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:47.224517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.225097 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.225100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:47.225169 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.225205 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.225563 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.225569 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:38:47.225630 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 16:38:47.225687 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":125,"threads_started":1}
I20260925 16:38:47.232560 8332 raft_consensus.cc:1261] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.232604 8052 raft_consensus.cc:1261] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.233019 8437 consensus_queue.cc:1035] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.233242 8454 consensus_queue.cc:1035] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.241190 8420 mvcc.cc:205] Tried to move back new op lower bound from 7333291324340506624 to 7333291323856130048. Current Snapshot: MvccSnapshot[applied={T|T < 7333291324340506624}]
I20260925 16:38:47.272221 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.272395 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.274386 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Sending DeleteTablet for 3 replicas of tablet 1944188667ae41f4ba3bdc7d86f14c1f
I20260925 16:38:47.274933 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 16:38:47.275044 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:43660
I20260925 16:38:47.275141 8431 tablet_replica.cc:330] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: stopping tablet replica
I20260925 16:38:47.275138 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet 1944188667ae41f4ba3bdc7d86f14c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:39988
I20260925 16:38:47.275207 8430 tablet_replica.cc:330] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: stopping tablet replica
I20260925 16:38:47.275282 8431 raft_consensus.cc:2229] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.275326 8430 raft_consensus.cc:2229] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.275348 8432 tablet_replica.cc:330] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: stopping tablet replica
I20260925 16:38:47.275408 8431 raft_consensus.cc:2258] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.275439 8430 raft_consensus.cc:2258] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.275446 8432 raft_consensus.cc:2229] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:47.275722 8432 raft_consensus.cc:2258] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.276437 8432 ts_tablet_manager.cc:1905] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.276437 8431 ts_tablet_manager.cc:1905] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.276459 8430 ts_tablet_manager.cc:1905] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.279645 8432 ts_tablet_manager.cc:1918] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.279723 8432 log.cc:1196] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/1944188667ae41f4ba3bdc7d86f14c1f
I20260925 16:38:47.279963 8430 ts_tablet_manager.cc:1918] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.279987 8432 ts_tablet_manager.cc:1939] T 1944188667ae41f4ba3bdc7d86f14c1f P 1f977546aa724138ae3c29a7fdd347a1: Deleting consensus metadata
I20260925 16:38:47.280035 8430 log.cc:1196] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/1944188667ae41f4ba3bdc7d86f14c1f
I20260925 16:38:47.280298 8430 ts_tablet_manager.cc:1939] T 1944188667ae41f4ba3bdc7d86f14c1f P 12d52b977302476298d5bf38bb3e6ff0: Deleting consensus metadata
I20260925 16:38:47.280738 7881 catalog_manager.cc:4841] TS 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605): tablet 1944188667ae41f4ba3bdc7d86f14c1f (table test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]) successfully deleted
I20260925 16:38:47.281549 7881 catalog_manager.cc:4841] TS 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721): tablet 1944188667ae41f4ba3bdc7d86f14c1f (table test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]) successfully deleted
I20260925 16:38:47.283252 8431 ts_tablet_manager.cc:1918] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.283357 8431 log.cc:1196] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/1944188667ae41f4ba3bdc7d86f14c1f
I20260925 16:38:47.283584 8431 ts_tablet_manager.cc:1939] T 1944188667ae41f4ba3bdc7d86f14c1f P 08ce6d188fde4443a15b46a858ee8ac6: Deleting consensus metadata
I20260925 16:38:47.284574 7882 catalog_manager.cc:4841] TS 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423): tablet 1944188667ae41f4ba3bdc7d86f14c1f (table test-table [id=cf7150a4057a4351a1e3e6c0ef7917ff]) successfully deleted
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:47.308183 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:47.301565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.0.97.254:44313)
0925 16:38:47.301699 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.301705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.301899 (+ 194us) server_negotiation.cc:407] Connection header received
0925 16:38:47.302077 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.302081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.302132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.302223 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:47.302960 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.303493 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.304392 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.304752 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.304866 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.308048 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35420: BlockingRecv error: recv got EOF from 127.0.0.1:35420 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM
I20260925 16:38:47.318221 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.309767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.97.254:44313)
0925 16:38:47.309891 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.309895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.310095 (+ 200us) server_negotiation.cc:407] Connection header received
0925 16:38:47.310246 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.310249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.310292 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.310391 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:47.310970 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.311578 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.312368 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.312623 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.312736 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.315088 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:47.315094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:47.315101 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:47.315125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:47.316535 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:47.316965 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:47.316969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:47.316971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:47.317052 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:47.317370 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:47.317373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:47.317375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:47.317552 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:47.317699 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.317942 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.318054 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:38:47.321411 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:38:47.321947 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:47.326678 8032 tablet_service.cc:1467] Processing CreateTablet for tablet dc41dc53061f4b99a4701d0a223322b8 (DEFAULT_TABLE table=test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.327049 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc41dc53061f4b99a4701d0a223322b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.327108 8293 tablet_service.cc:1467] Processing CreateTablet for tablet dc41dc53061f4b99a4701d0a223322b8 (DEFAULT_TABLE table=test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.327165 8172 tablet_service.cc:1467] Processing CreateTablet for tablet dc41dc53061f4b99a4701d0a223322b8 (DEFAULT_TABLE table=test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.327381 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc41dc53061f4b99a4701d0a223322b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.327469 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc41dc53061f4b99a4701d0a223322b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.329583 8408 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Bootstrap starting.
I20260925 16:38:47.329770 8406 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Bootstrap starting.
I20260925 16:38:47.329988 8407 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Bootstrap starting.
I20260925 16:38:47.330413 8408 tablet_bootstrap.cc:654] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.330616 8406 tablet_bootstrap.cc:654] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.330924 8407 tablet_bootstrap.cc:654] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.331521 8408 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: No bootstrap required, opened a new log
I20260925 16:38:47.331616 8408 ts_tablet_manager.cc:1397] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:38:47.332118 8407 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: No bootstrap required, opened a new log
I20260925 16:38:47.332077 8408 raft_consensus.cc:348] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.332182 8408 raft_consensus.cc:374] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.332195 8407 ts_tablet_manager.cc:1397] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:47.332218 8408 raft_consensus.cc:729] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.332356 8408 consensus_queue.cc:260] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.332620 8408 ts_tablet_manager.cc:1428] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:47.332618 8407 raft_consensus.cc:348] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.332718 8407 raft_consensus.cc:374] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.332757 8407 raft_consensus.cc:729] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.332887 8407 consensus_queue.cc:260] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.333164 8407 ts_tablet_manager.cc:1428] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:47.333978 8406 tablet_bootstrap.cc:492] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: No bootstrap required, opened a new log
I20260925 16:38:47.334085 8406 ts_tablet_manager.cc:1397] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 16:38:47.334534 8406 raft_consensus.cc:348] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.334640 8406 raft_consensus.cc:374] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.334673 8406 raft_consensus.cc:729] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d52b977302476298d5bf38bb3e6ff0, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.334825 8406 consensus_queue.cc:260] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.335106 8406 ts_tablet_manager.cc:1428] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 16:38:47.337065 8417 raft_consensus.cc:482] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:47.337181 8417 raft_consensus.cc:504] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.337584 8417 leader_election.cc:288] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.337913 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc41dc53061f4b99a4701d0a223322b8" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f977546aa724138ae3c29a7fdd347a1" is_pre_election: true
I20260925 16:38:47.338004 8191 raft_consensus.cc:2454] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 0.
I20260925 16:38:47.338265 7985 leader_election.cc:302] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.338367 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc41dc53061f4b99a4701d0a223322b8" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0" is_pre_election: true
I20260925 16:38:47.338423 8417 raft_consensus.cc:2781] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:47.338467 8417 raft_consensus.cc:482] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:47.338511 8417 raft_consensus.cc:3037] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.338495 8332 raft_consensus.cc:2454] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 0.
I20260925 16:38:47.339469 8417 raft_consensus.cc:504] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.339825 8417 leader_election.cc:288] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.340109 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc41dc53061f4b99a4701d0a223322b8" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f977546aa724138ae3c29a7fdd347a1"
I20260925 16:38:47.340224 8191 raft_consensus.cc:3037] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.340162 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc41dc53061f4b99a4701d0a223322b8" candidate_uuid: "08ce6d188fde4443a15b46a858ee8ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0"
I20260925 16:38:47.340302 8332 raft_consensus.cc:3037] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.341356 8191 raft_consensus.cc:2454] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 1.
I20260925 16:38:47.341463 8332 raft_consensus.cc:2454] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ce6d188fde4443a15b46a858ee8ac6 in term 1.
I20260925 16:38:47.341642 7985 leader_election.cc:302] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ce6d188fde4443a15b46a858ee8ac6, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.341835 8417 raft_consensus.cc:2781] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:47.341931 8417 raft_consensus.cc:686] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 LEADER]: Becoming Leader. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Running, Role: LEADER
I20260925 16:38:47.342053 8417 consensus_queue.cc:237] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.343631 7894 catalog_manager.cc:5495] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "08ce6d188fde4443a15b46a858ee8ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:47.354823 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.351588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.0.97.193:41423)
0925 16:38:47.351727 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.351731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.351744 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:47.351905 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.351908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.351962 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.352048 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:47.352284 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.352795 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.353750 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.353991 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.354103 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.354348 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.354428 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.354509 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.354625 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.354679 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:38:47.356832 8332 raft_consensus.cc:1261] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Refusing update from remote peer 08ce6d188fde4443a15b46a858ee8ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.356911 8191 raft_consensus.cc:1261] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Refusing update from remote peer 08ce6d188fde4443a15b46a858ee8ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.357347 8417 consensus_queue.cc:1035] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.357496 8425 consensus_queue.cc:1035] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.366142 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.366300 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.368172 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Sending DeleteTablet for 3 replicas of tablet dc41dc53061f4b99a4701d0a223322b8
I20260925 16:38:47.368903 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet dc41dc53061f4b99a4701d0a223322b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 16:38:47.368919 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet dc41dc53061f4b99a4701d0a223322b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:43660
I20260925 16:38:47.368919 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet dc41dc53061f4b99a4701d0a223322b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:39988
I20260925 16:38:47.369079 8431 tablet_replica.cc:330] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: stopping tablet replica
I20260925 16:38:47.369079 8430 tablet_replica.cc:330] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: stopping tablet replica
I20260925 16:38:47.369189 8431 raft_consensus.cc:2229] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:47.369187 8430 raft_consensus.cc:2229] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.369334 8430 raft_consensus.cc:2258] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.369387 8431 raft_consensus.cc:2258] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.369549 8432 tablet_replica.cc:330] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: stopping tablet replica
I20260925 16:38:47.369674 8432 raft_consensus.cc:2229] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.369840 8432 raft_consensus.cc:2258] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.370326 8430 ts_tablet_manager.cc:1905] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.370482 8431 ts_tablet_manager.cc:1905] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.370482 8432 ts_tablet_manager.cc:1905] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.372700 8432 ts_tablet_manager.cc:1918] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.372701 8431 ts_tablet_manager.cc:1918] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.372712 8430 ts_tablet_manager.cc:1918] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:38:47.372768 8432 log.cc:1196] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/dc41dc53061f4b99a4701d0a223322b8
I20260925 16:38:47.372768 8431 log.cc:1196] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/dc41dc53061f4b99a4701d0a223322b8
I20260925 16:38:47.372788 8430 log.cc:1196] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/dc41dc53061f4b99a4701d0a223322b8
I20260925 16:38:47.373044 8430 ts_tablet_manager.cc:1939] T dc41dc53061f4b99a4701d0a223322b8 P 12d52b977302476298d5bf38bb3e6ff0: Deleting consensus metadata
I20260925 16:38:47.373045 8431 ts_tablet_manager.cc:1939] T dc41dc53061f4b99a4701d0a223322b8 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting consensus metadata
I20260925 16:38:47.373045 8432 ts_tablet_manager.cc:1939] T dc41dc53061f4b99a4701d0a223322b8 P 1f977546aa724138ae3c29a7fdd347a1: Deleting consensus metadata
I20260925 16:38:47.373900 7881 catalog_manager.cc:4841] TS 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721): tablet dc41dc53061f4b99a4701d0a223322b8 (table test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]) successfully deleted
I20260925 16:38:47.374073 7882 catalog_manager.cc:4841] TS 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423): tablet dc41dc53061f4b99a4701d0a223322b8 (table test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]) successfully deleted
I20260925 16:38:47.374176 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:38:47.374353 7881 catalog_manager.cc:4841] TS 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605): tablet dc41dc53061f4b99a4701d0a223322b8 (table test-table [id=9c03ee7d435f442fb1ec00cea2bb9aad]) successfully deleted
W20260925 16:38:47.374629 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:38:47.379182 8032 tablet_service.cc:1467] Processing CreateTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 (DEFAULT_TABLE table=test-table [id=181e5bd779f040528e9ad7b45a624962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.379356 8172 tablet_service.cc:1467] Processing CreateTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 (DEFAULT_TABLE table=test-table [id=181e5bd779f040528e9ad7b45a624962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.379488 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35009bdd02d461db7f5b1105aa3eb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.379447 8293 tablet_service.cc:1467] Processing CreateTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 (DEFAULT_TABLE table=test-table [id=181e5bd779f040528e9ad7b45a624962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:38:47.379617 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35009bdd02d461db7f5b1105aa3eb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.379657 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35009bdd02d461db7f5b1105aa3eb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:47.381855 8407 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Bootstrap starting.
I20260925 16:38:47.381883 8408 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Bootstrap starting.
I20260925 16:38:47.382578 8407 tablet_bootstrap.cc:654] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.382695 8408 tablet_bootstrap.cc:654] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.383147 8406 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Bootstrap starting.
I20260925 16:38:47.383706 8407 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: No bootstrap required, opened a new log
I20260925 16:38:47.383786 8407 ts_tablet_manager.cc:1397] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:47.384057 8406 tablet_bootstrap.cc:654] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:47.384198 8407 raft_consensus.cc:348] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.384320 8407 raft_consensus.cc:374] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.384361 8407 raft_consensus.cc:729] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.384469 8407 consensus_queue.cc:260] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.384750 8407 ts_tablet_manager.cc:1428] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:47.385970 8408 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: No bootstrap required, opened a new log
I20260925 16:38:47.386052 8408 ts_tablet_manager.cc:1397] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 16:38:47.386423 8408 raft_consensus.cc:348] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.386531 8408 raft_consensus.cc:374] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.386564 8408 raft_consensus.cc:729] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce6d188fde4443a15b46a858ee8ac6, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.386680 8408 consensus_queue.cc:260] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.386920 8408 ts_tablet_manager.cc:1428] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:38:47.387588 8406 tablet_bootstrap.cc:492] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: No bootstrap required, opened a new log
I20260925 16:38:47.387665 8406 ts_tablet_manager.cc:1397] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260925 16:38:47.388054 8406 raft_consensus.cc:348] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.388157 8406 raft_consensus.cc:374] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:47.388188 8406 raft_consensus.cc:729] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d52b977302476298d5bf38bb3e6ff0, State: Initialized, Role: FOLLOWER
I20260925 16:38:47.388307 8406 consensus_queue.cc:260] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.388557 8406 ts_tablet_manager.cc:1428] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:38:47.418274 8454 raft_consensus.cc:482] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:38:47.418414 8454 raft_consensus.cc:504] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.418838 8454 leader_election.cc:288] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.419283 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35009bdd02d461db7f5b1105aa3eb06" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0" is_pre_election: true
I20260925 16:38:47.419348 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35009bdd02d461db7f5b1105aa3eb06" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6" is_pre_election: true
I20260925 16:38:47.419448 8332 raft_consensus.cc:2454] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 0.
I20260925 16:38:47.419497 8052 raft_consensus.cc:2454] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 0.
I20260925 16:38:47.419802 8125 leader_election.cc:302] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d52b977302476298d5bf38bb3e6ff0, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.419976 8454 raft_consensus.cc:2781] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:38:47.420053 8454 raft_consensus.cc:482] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:38:47.420101 8454 raft_consensus.cc:3037] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.421310 8454 raft_consensus.cc:504] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.421676 8454 leader_election.cc:288] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Requested vote from peers 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423), 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721)
I20260925 16:38:47.421940 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35009bdd02d461db7f5b1105aa3eb06" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ce6d188fde4443a15b46a858ee8ac6"
I20260925 16:38:47.422037 8052 raft_consensus.cc:3037] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.422021 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35009bdd02d461db7f5b1105aa3eb06" candidate_uuid: "1f977546aa724138ae3c29a7fdd347a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d52b977302476298d5bf38bb3e6ff0"
I20260925 16:38:47.422127 8332 raft_consensus.cc:3037] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:47.423236 8052 raft_consensus.cc:2454] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.423245 8332 raft_consensus.cc:2454] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f977546aa724138ae3c29a7fdd347a1 in term 1.
I20260925 16:38:47.423503 8125 leader_election.cc:302] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d52b977302476298d5bf38bb3e6ff0, 1f977546aa724138ae3c29a7fdd347a1; no voters:
I20260925 16:38:47.423660 8454 raft_consensus.cc:2781] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:47.423754 8454 raft_consensus.cc:686] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1f977546aa724138ae3c29a7fdd347a1, State: Running, Role: LEADER
I20260925 16:38:47.423878 8454 consensus_queue.cc:237] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } }
I20260925 16:38:47.425556 7894 catalog_manager.cc:5495] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "1f977546aa724138ae3c29a7fdd347a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f977546aa724138ae3c29a7fdd347a1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 } health_report { overall_health: UNKNOWN } } }
I20260925 16:38:47.447732 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:47.444618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.97.194:43605)
0925 16:38:47.444952 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:47.444955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:47.444965 (+ 10us) server_negotiation.cc:407] Connection header received
0925 16:38:47.444997 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:47.444999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:47.445034 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:47.445120 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:38:47.445385 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.445892 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.446707 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:47.446940 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:47.447051 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:47.447291 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:38:47.447366 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:38:47.447448 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:47.447536 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 16:38:47.447580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260925 16:38:47.453956 8332 raft_consensus.cc:1261] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.454110 8051 raft_consensus.cc:1261] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Refusing update from remote peer 1f977546aa724138ae3c29a7fdd347a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:38:47.454458 8437 consensus_queue.cc:1035] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d52b977302476298d5bf38bb3e6ff0" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.454795 8437 consensus_queue.cc:1035] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ce6d188fde4443a15b46a858ee8ac6" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:38:47.489753 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.489931 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:38:47.491971 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a61b35a0e034a23b29a1d959bc358d4: Sending DeleteTablet for 3 replicas of tablet d35009bdd02d461db7f5b1105aa3eb06
I20260925 16:38:47.492522 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 16:38:47.492718 8431 tablet_replica.cc:330] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: stopping tablet replica
I20260925 16:38:47.492872 8431 raft_consensus.cc:2229] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.493010 8431 raft_consensus.cc:2258] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.493363 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:39988
I20260925 16:38:47.493544 8432 tablet_replica.cc:330] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: stopping tablet replica
I20260925 16:38:47.493640 8432 raft_consensus.cc:2229] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:38:47.493718 8431 ts_tablet_manager.cc:1905] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.493831 8432 raft_consensus.cc:2258] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.495821 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet d35009bdd02d461db7f5b1105aa3eb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:38:47 UTC) from {username='oryx', principal='oryx/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:43660
I20260925 16:38:47.495918 8431 ts_tablet_manager.cc:1918] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.495986 8431 log.cc:1196] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/d35009bdd02d461db7f5b1105aa3eb06
I20260925 16:38:47.496188 8430 tablet_replica.cc:330] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: stopping tablet replica
I20260925 16:38:47.496269 8431 ts_tablet_manager.cc:1939] T d35009bdd02d461db7f5b1105aa3eb06 P 08ce6d188fde4443a15b46a858ee8ac6: Deleting consensus metadata
I20260925 16:38:47.496320 8430 raft_consensus.cc:2229] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:38:47.496444 8430 raft_consensus.cc:2258] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:38:47.496973 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7967
I20260925 16:38:47.496940 7882 catalog_manager.cc:4841] TS 08ce6d188fde4443a15b46a858ee8ac6 (127.0.97.193:41423): tablet d35009bdd02d461db7f5b1105aa3eb06 (table test-table [id=181e5bd779f040528e9ad7b45a624962]) successfully deleted
I20260925 16:38:47.497083 8432 ts_tablet_manager.cc:1905] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.499694 8430 ts_tablet_manager.cc:1905] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:38:47.499923 8432 ts_tablet_manager.cc:1918] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.500017 8432 log.cc:1196] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/d35009bdd02d461db7f5b1105aa3eb06
I20260925 16:38:47.500290 8432 ts_tablet_manager.cc:1939] T d35009bdd02d461db7f5b1105aa3eb06 P 1f977546aa724138ae3c29a7fdd347a1: Deleting consensus metadata
I20260925 16:38:47.501039 7881 catalog_manager.cc:4841] TS 1f977546aa724138ae3c29a7fdd347a1 (127.0.97.194:43605): tablet d35009bdd02d461db7f5b1105aa3eb06 (table test-table [id=181e5bd779f040528e9ad7b45a624962]) successfully deleted
I20260925 16:38:47.501845 8430 ts_tablet_manager.cc:1918] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:38:47.501950 8430 log.cc:1196] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/d35009bdd02d461db7f5b1105aa3eb06
I20260925 16:38:47.502179 8430 ts_tablet_manager.cc:1939] T d35009bdd02d461db7f5b1105aa3eb06 P 12d52b977302476298d5bf38bb3e6ff0: Deleting consensus metadata
I20260925 16:38:47.502928 7881 catalog_manager.cc:4841] TS 12d52b977302476298d5bf38bb3e6ff0 (127.0.97.195:42721): tablet d35009bdd02d461db7f5b1105aa3eb06 (table test-table [id=181e5bd779f040528e9ad7b45a624962]) successfully deleted
I20260925 16:38:47.507917 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8107
I20260925 16:38:47.515951 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8247
I20260925 16:38:47.521842 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 7863
2026-09-25T16:38:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2716 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[8490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[8490](info): set up 2 sockets
Sep 25 16:38:47 dist-test-slave-kxl8 krb5kdc[8490](info): commencing operation
krb5kdc: starting...
W20260925 16:38:48.570194 391 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:48 dist-test-slave-kxl8 krb5kdc[8490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:48.692745 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36293
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:48.802039 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:48.802340 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:48.802421 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:48.806460 8503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:48.806545 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:48.806586 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:48.806644 8503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:48.806680 8503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:48.811959 8503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:48.813023 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:48.814059 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:48.822726 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:48.822726 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:48.822952 8503 server_base.cc:1034] running on GCE node
W20260925 16:38:48.823050 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:48.823587 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:48.824693 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:48.825857 8503 hybrid_clock.cc:648] HybridClock initialized: now 1790354328825840 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T16:38:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1307 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:38:48 dist-test-slave-kxl8 krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:38:48 dist-test-slave-kxl8 krb5kdc[8516](info): set up 2 sockets
Sep 25 16:38:48 dist-test-slave-kxl8 krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260925 16:38:49.897984 391 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:38:49 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:38:50.020622 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39025
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:50.131870 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:50.132169 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:50.132265 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:50.136809 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:50.136875 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:50.136897 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:50.136960 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:50.136996 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:50.142620 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:50.143729 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:50.144775 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:50.153687 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.153687 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.153821 8529 server_base.cc:1034] running on GCE node
W20260925 16:38:50.153637 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.154378 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:50.155481 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:50.156659 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790354330156632 us; error 39 us; skew 500 ppm
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:50.159493 8529 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:50.160440 8529 webserver.cc:460] Webserver started at http://127.0.97.254:34395/ using document root <none> and password file <none>
I20260925 16:38:50.161002 8529 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:50.161062 8529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:50.161332 8529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:50.163359 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "3cb9aeb78bba4eccb48b910d14a37830"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "2324475f9eff4f2cccb4a556958df8a8"
server_key_iv: "aaad63d4331f8117b6b445d25f9f2944"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.163964 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "3cb9aeb78bba4eccb48b910d14a37830"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "2324475f9eff4f2cccb4a556958df8a8"
server_key_iv: "aaad63d4331f8117b6b445d25f9f2944"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.167702 8529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:38:50.170318 8544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.171808 8529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:50.171972 8529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "3cb9aeb78bba4eccb48b910d14a37830"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "2324475f9eff4f2cccb4a556958df8a8"
server_key_iv: "aaad63d4331f8117b6b445d25f9f2944"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.172101 8529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:50.192533 8529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:50.195629 8529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:50.195838 8529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:50.203184 8529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:39025
I20260925 16:38:50.203202 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:39025 every 8 connection(s)
I20260925 16:38:50.204198 8529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:50.206934 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8529
I20260925 16:38:50.207064 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:38:50.207379 391 external_mini_cluster.cc:1428] Setting key 090e6d75b4d56506e69e8f7cbfa7d282
I20260925 16:38:50.207537 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:38:50.214696 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Bootstrap starting.
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:50.217581 8597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Neither blocks nor log segments found. Creating new log.
I20260925 16:38:50.218494 8597 log.cc:824] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:50.220811 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: No bootstrap required, opened a new log
I20260925 16:38:50.221665 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.208790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.97.254:39025)
0925 16:38:50.209206 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:50.209221 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:50.209255 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:38:50.209946 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:50.209962 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:50.210277 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:50.210615 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:50.211387 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.212374 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.213233 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.213759 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.213946 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.216873 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:50.216878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:50.216890 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:50.216915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:50.218793 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.219270 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.219283 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.219289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.219394 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.219703 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.219705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.219706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.220115 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:50.220248 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:50.220595 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 16:38:50.220876 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260925 16:38:50.224562 8597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:50.224867 8597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:38:50.224962 8597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb9aeb78bba4eccb48b910d14a37830, State: Initialized, Role: FOLLOWER
I20260925 16:38:50.225474 8597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:50.225647 8597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:50.225741 8597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:50.225878 8597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:38:50.226815 8597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:50.227166 8597 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb9aeb78bba4eccb48b910d14a37830; no voters:
I20260925 16:38:50.227497 8597 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:38:50.227591 8602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:38:50.227861 8602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb9aeb78bba4eccb48b910d14a37830, State: Running, Role: LEADER
I20260925 16:38:50.228530 8602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:50.228974 8597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:50.230321 8604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb9aeb78bba4eccb48b910d14a37830. Latest consensus state: current_term: 1 leader_uuid: "3cb9aeb78bba4eccb48b910d14a37830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } } }
I20260925 16:38:50.230320 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cb9aeb78bba4eccb48b910d14a37830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } } }
I20260925 16:38:50.230484 8604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:50.230484 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:50.230966 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:50.233767 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:50.239346 8611 catalog_manager.cc:1347] Generated new cluster ID: ce196c2da3ee4d8e90bac63ea55904c8
I20260925 16:38:50.239411 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:50.262151 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:38:50.263083 8611 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:50.273322 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Generated new TSK 0
I20260925 16:38:50.274041 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:38:50.305473 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:50.417028 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:50.417374 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:50.417491 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:50.421624 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:50.421705 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:50.421794 8623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:50.426972 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:50.428007 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:50.429109 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:50.439590 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.439592 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.439592 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.439647 8623 server_base.cc:1034] running on GCE node
I20260925 16:38:50.440243 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:50.440848 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:50.442011 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790354330441992 us; error 31 us; skew 500 ppm
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:50.444816 8623 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:38:50.445675 8623 webserver.cc:460] Webserver started at http://127.0.97.193:45307/ using document root <none> and password file <none>
I20260925 16:38:50.446173 8623 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:50.446229 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:50.446396 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:50.448627 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "94ebb52d13f845d893ca56be4fd3430b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "cb062a327efef54cb132640e02b40844"
server_key_iv: "43319872e08a9ef53fa576febba50e6a"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.449227 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "94ebb52d13f845d893ca56be4fd3430b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "cb062a327efef54cb132640e02b40844"
server_key_iv: "43319872e08a9ef53fa576febba50e6a"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.452891 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:38:50.455348 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.456601 8623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:50.456696 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "94ebb52d13f845d893ca56be4fd3430b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "cb062a327efef54cb132640e02b40844"
server_key_iv: "43319872e08a9ef53fa576febba50e6a"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.456782 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:50.475167 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:50.478654 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:50.478811 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:50.479367 8623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:50.480125 8623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:50.480172 8623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.480211 8623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:50.480225 8623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.489059 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:37075
I20260925 16:38:50.489231 8751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:37075 every 8 connection(s)
I20260925 16:38:50.489996 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:50.492338 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8623
I20260925 16:38:50.492465 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:38:50.492772 391 external_mini_cluster.cc:1428] Setting key e12c001854d4df669b184e24289e226e
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:38:50.505409 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.491888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:49789 (local address 127.0.97.254:39025)
0925 16:38:50.492043 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:50.492047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:50.492874 (+ 827us) server_negotiation.cc:407] Connection header received
0925 16:38:50.494478 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:50.494482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:50.494530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:50.494606 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:50.495965 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.496506 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.497801 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.498033 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.498145 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.500741 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:50.500750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:50.500752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:50.500774 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:50.502348 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.503241 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.503248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.503252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.503360 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.503899 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.503904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.503907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.504171 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:50.504292 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:50.504970 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 16:38:50.505131 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:38:50.506488 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.492236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:49789)
0925 16:38:50.492712 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:50.492764 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:50.494278 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:50.494792 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:50.494799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:50.495157 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:50.495792 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.495811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.496679 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.496682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.497660 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.497668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.498183 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.498188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.498353 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.498950 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:50.498966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:50.500578 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:50.502494 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.502500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.502515 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.503111 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.503466 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.503469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.503471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.503768 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.504291 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:50.504298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:50.504644 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 16:38:50.504915 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 16:38:50.505328 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":145,"threads_started":1}
I20260925 16:38:50.508665 8752 heartbeater.cc:347] Connected to a master server at 127.0.97.254:39025
I20260925 16:38:50.508958 8752 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:50.509824 8752 heartbeater.cc:511] Master 127.0.97.254:39025 requested a full tablet report, sending...
I20260925 16:38:50.511904 8561 ts_manager.cc:194] Registered new tserver with Master: 94ebb52d13f845d893ca56be4fd3430b (127.0.97.193:37075)
I20260925 16:38:50.513670 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:49789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:38:50.525250 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:50.634505 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:50.634758 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:50.634801 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:50.638826 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:50.638883 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:50.638983 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:50.644140 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:50.645181 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:50.646284 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:50.656315 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.656370 8757 server_base.cc:1034] running on GCE node
W20260925 16:38:50.656320 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.656303 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.657001 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:50.657699 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:50.658922 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790354330658902 us; error 31 us; skew 500 ppm
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:50.662120 8757 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:38:50.663000 8757 webserver.cc:460] Webserver started at http://127.0.97.194:33805/ using document root <none> and password file <none>
I20260925 16:38:50.663571 8757 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:50.663667 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:50.663923 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:50.666010 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "559693fe34844f75b6c6d7fb4a8b0e4b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "9fe4beb282d91b00895d3034713d3d2e"
server_key_iv: "5183d1a4412ceab26580d5b540ad2533"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.666563 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "559693fe34844f75b6c6d7fb4a8b0e4b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "9fe4beb282d91b00895d3034713d3d2e"
server_key_iv: "5183d1a4412ceab26580d5b540ad2533"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.670272 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:50.672688 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.674100 8757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:50.674237 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "559693fe34844f75b6c6d7fb4a8b0e4b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "9fe4beb282d91b00895d3034713d3d2e"
server_key_iv: "5183d1a4412ceab26580d5b540ad2533"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.674350 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:50.693701 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:50.696914 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:50.697120 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:50.697744 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:50.698544 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:50.698617 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.698693 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:50.698738 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.707564 8757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:40583
I20260925 16:38:50.707592 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:40583 every 8 connection(s)
I20260925 16:38:50.708444 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:50.712071 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8757
I20260925 16:38:50.712208 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:38:50.712493 391 external_mini_cluster.cc:1428] Setting key b5ce9498a8f3312aa3771a1e5b171704
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:38:50.723565 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.710232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:55533 (local address 127.0.97.254:39025)
0925 16:38:50.710385 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:50.710388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:50.711135 (+ 747us) server_negotiation.cc:407] Connection header received
0925 16:38:50.712919 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:50.712923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:50.712982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:50.713065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:50.714691 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.715196 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.716474 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.716880 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.717000 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.719836 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:50.719840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:50.719843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:50.719888 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:50.721580 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.722071 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.722075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.722077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.722168 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.722500 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.722506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.722509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.722747 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:50.722859 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:50.723281 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 16:38:50.723381 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:38:50.724325 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.710582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:55533)
0925 16:38:50.710990 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:50.711031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:50.712626 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:50.713422 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:50.713431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:50.713887 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:50.714541 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.714563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.715320 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.715327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.716351 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.716360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.717025 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.717028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.717163 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.717831 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:50.717847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:50.719666 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:50.721690 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.721695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.721706 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.721963 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.722259 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.722261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.722262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.722399 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.722859 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:50.722864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:50.723074 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 16:38:50.723225 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 16:38:50.723473 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:38:50.725683 8886 heartbeater.cc:347] Connected to a master server at 127.0.97.254:39025
I20260925 16:38:50.725940 8886 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:50.726410 8886 heartbeater.cc:511] Master 127.0.97.254:39025 requested a full tablet report, sending...
I20260925 16:38:50.727522 8561 ts_manager.cc:194] Registered new tserver with Master: 559693fe34844f75b6c6d7fb4a8b0e4b (127.0.97.194:40583)
I20260925 16:38:50.728075 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:55533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:38:50.741084 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:50.850279 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:50.850571 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:50.850651 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:50.854947 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:50.855028 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:50.855113 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:50.861083 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:50.862429 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:50.863631 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:50.873558 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.873553 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:50.873570 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:50.873878 8891 server_base.cc:1034] running on GCE node
I20260925 16:38:50.874279 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:50.874948 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:50.876140 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790354330876123 us; error 38 us; skew 500 ppm
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:50.878930 8891 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:38:50.879755 8891 webserver.cc:460] Webserver started at http://127.0.97.195:45101/ using document root <none> and password file <none>
I20260925 16:38:50.880300 8891 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:50.880406 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:50.880635 8891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:38:50.882643 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "0385a32923174fc8800ec57f0146143b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "0791a3200cf1a8965c73d40878efc510"
server_key_iv: "b807772536a01d0c6243fec69b3dee29"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.883194 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "0385a32923174fc8800ec57f0146143b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "0791a3200cf1a8965c73d40878efc510"
server_key_iv: "b807772536a01d0c6243fec69b3dee29"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.886926 8891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 16:38:50.889246 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.890492 8891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:50.890625 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "0385a32923174fc8800ec57f0146143b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "0791a3200cf1a8965c73d40878efc510"
server_key_iv: "b807772536a01d0c6243fec69b3dee29"
server_key_version: "encryptionkey@0"
I20260925 16:38:50.890738 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:50.903198 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:50.906023 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:50.906219 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:50.906783 8891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:50.907563 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:50.907644 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.907716 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:50.907752 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:50.916790 8891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:36795
I20260925 16:38:50.916815 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:36795 every 8 connection(s)
I20260925 16:38:50.917677 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
Sep 25 16:38:50 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:38:50.927088 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 8891
I20260925 16:38:50.927253 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:38:50.927542 391 external_mini_cluster.cc:1428] Setting key 2dbb890a26db82bc7659fe2252c5ef3a
I20260925 16:38:50.930851 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.919352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:48995 (local address 127.0.97.254:39025)
0925 16:38:50.919528 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:50.919533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:50.920289 (+ 756us) server_negotiation.cc:407] Connection header received
0925 16:38:50.921647 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:50.921650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:50.921695 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:50.921767 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:50.922863 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.923384 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.924455 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.924655 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.924727 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.927182 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:50.927188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:50.927191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:50.927221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:50.928708 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.929270 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.929296 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.929298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.929388 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:50.929732 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:50.929735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:50.929737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:50.929965 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:50.930069 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:50.930577 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 16:38:50.930690 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:38:50.931627 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:50.919717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:48995)
0925 16:38:50.920133 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:50.920175 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:50.921453 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:50.921945 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:50.921952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:50.922282 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:50.922730 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.922742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.923488 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.923491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.924350 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:50.924359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:50.924774 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:50.924777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:50.924920 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:50.925488 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:50.925503 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:50.926981 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:38:50.928895 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.928900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.928913 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.929161 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.929485 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:38:50.929489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:38:50.929492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:38:50.929623 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:38:50.930066 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:38:50.930072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:38:50.930335 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 16:38:50.930523 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 16:38:50.930813 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260925 16:38:50.932596 9020 heartbeater.cc:347] Connected to a master server at 127.0.97.254:39025
I20260925 16:38:50.932827 9020 heartbeater.cc:464] Registering TS with master...
I20260925 16:38:50.933328 9020 heartbeater.cc:511] Master 127.0.97.254:39025 requested a full tablet report, sending...
I20260925 16:38:50.934263 8561 ts_manager.cc:194] Registered new tserver with Master: 0385a32923174fc8800ec57f0146143b (127.0.97.195:36795)
I20260925 16:38:50.934758 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:48995
I20260925 16:38:50.941043 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
Entry for principal oryx/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab.
WARNING: no policy specified for oryx/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
Entry for principal oryx/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab.
WARNING: no policy specified for oryx/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
Entry for principal oryx/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab.
I20260925 16:38:51.016623 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8623
I20260925 16:38:51.022351 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8757
I20260925 16:38:51.028369 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8891
I20260925 16:38:51.033715 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 8529
I20260925 16:38:51.039758 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39025
--webserver_interface=127.0.97.254
--webserver_port=34395
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:38:51.149199 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:51.149544 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:51.149622 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:51.153932 9029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:38:51.154032 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:51.154070 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:51.154107 9029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:38:51.154137 9029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:38:51.159448 9029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:51.160521 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:51.161573 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:51.170238 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.170251 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.170257 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:51.170543 9029 server_base.cc:1034] running on GCE node
I20260925 16:38:51.171134 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:51.172252 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:51.173452 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790354331173426 us; error 40 us; skew 500 ppm
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:51.176213 9029 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:38:51.177114 9029 webserver.cc:460] Webserver started at http://127.0.97.254:34395/ using document root <none> and password file <none>
I20260925 16:38:51.177917 9029 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:51.178020 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:51.182487 9029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:38:51.184504 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.185842 9029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:51.185974 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "3cb9aeb78bba4eccb48b910d14a37830"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "2324475f9eff4f2cccb4a556958df8a8"
server_key_iv: "aaad63d4331f8117b6b445d25f9f2944"
server_key_version: "encryptionkey@0"
I20260925 16:38:51.186471 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:51.197690 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:51.200642 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:51.200805 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:51.207719 9029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:39025
I20260925 16:38:51.207732 9096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:39025 every 8 connection(s)
I20260925 16:38:51.208806 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:38:51.212888 9097 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:38:51.214515 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Bootstrap starting.
I20260925 16:38:51.215212 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9029
I20260925 16:38:51.215657 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:37075
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:39025
--webserver_port=45307
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:38:51.236140 9097 log.cc:824] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Log is configured to *not* fsync() on all Append() calls
I20260925 16:38:51.244243 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:38:51.244643 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Bootstrap complete.
I20260925 16:38:51.250108 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:51.250615 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb9aeb78bba4eccb48b910d14a37830, State: Initialized, Role: FOLLOWER
I20260925 16:38:51.251175 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:51.251314 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:38:51.251382 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:38:51.251473 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:38:51.253938 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:51.254281 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb9aeb78bba4eccb48b910d14a37830; no voters:
I20260925 16:38:51.254604 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:38:51.254815 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:38:51.255131 9101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [term 2 LEADER]: Becoming Leader. State: Replica: 3cb9aeb78bba4eccb48b910d14a37830, State: Running, Role: LEADER
I20260925 16:38:51.255502 9101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } }
I20260925 16:38:51.255844 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:38:51.257083 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb9aeb78bba4eccb48b910d14a37830. Latest consensus state: current_term: 2 leader_uuid: "3cb9aeb78bba4eccb48b910d14a37830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } } }
I20260925 16:38:51.257228 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:51.257126 9102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3cb9aeb78bba4eccb48b910d14a37830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb9aeb78bba4eccb48b910d14a37830" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39025 } } }
I20260925 16:38:51.257531 9102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830 [sys.catalog]: This master's current role is: LEADER
I20260925 16:38:51.260517 9116 catalog_manager.cc:1259] Loaded cluster ID: ce196c2da3ee4d8e90bac63ea55904c8
I20260925 16:38:51.260593 9116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: loading cluster ID for follower catalog manager: success
I20260925 16:38:51.262427 9116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: acquiring CA information for follower catalog manager: success
I20260925 16:38:51.263497 9116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:38:51.263628 9111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:38:51.264279 9111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:38:51.264683 9111 catalog_manager.cc:1259] Loaded cluster ID: ce196c2da3ee4d8e90bac63ea55904c8
I20260925 16:38:51.264755 9111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:38:51.265237 9111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:38:51.265662 9111 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3cb9aeb78bba4eccb48b910d14a37830: Loaded TSK: 0
I20260925 16:38:51.266228 9111 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:38:51.357193 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:51.357581 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:51.357666 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:51.361846 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:51.362083 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:51.362251 9099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:38:51.367795 9099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.193.keytab
--principal=oryx/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:37075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=45307
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:51.368786 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:51.369788 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:51.379082 9126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.379108 9124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.379123 9123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:51.379632 9099 server_base.cc:1034] running on GCE node
I20260925 16:38:51.380024 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:51.380625 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:51.381800 9099 hybrid_clock.cc:648] HybridClock initialized: now 1790354331381781 us; error 37 us; skew 500 ppm
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354331, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:51.384665 9099 init.cc:377] Logged in from keytab as oryx/127.0.97.193@KRBTEST.COM (short username oryx)
I20260925 16:38:51.385689 9099 webserver.cc:460] Webserver started at http://127.0.97.193:45307/ using document root <none> and password file <none>
I20260925 16:38:51.386230 9099 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:51.386298 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:51.390138 9099 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:38:51.392251 9133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.393571 9099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:51.393656 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "94ebb52d13f845d893ca56be4fd3430b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "cb062a327efef54cb132640e02b40844"
server_key_iv: "43319872e08a9ef53fa576febba50e6a"
server_key_version: "encryptionkey@0"
I20260925 16:38:51.394194 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:51.429060 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:51.432531 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:51.432752 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:51.433441 9099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:51.434337 9099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:51.434386 9099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.434455 9099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:51.434487 9099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.443632 9099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:37075
I20260925 16:38:51.443634 9246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:37075 every 8 connection(s)
I20260925 16:38:51.444694 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:38:51.454079 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9099
I20260925 16:38:51.454531 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:40583
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:39025
--webserver_port=33805
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:51.458796 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.446793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:43589 (local address 127.0.97.254:39025)
0925 16:38:51.447254 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.447274 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.447494 (+ 220us) server_negotiation.cc:407] Connection header received
0925 16:38:51.449031 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.449043 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.449446 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.449722 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.450987 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.451988 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.453427 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.453984 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.454163 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.458073 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:43589: BlockingRecv error: recv got EOF from 127.0.97.193:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":94,"threads_started":1}
I20260925 16:38:51.458901 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.446867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:43589)
0925 16:38:51.447342 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.447382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.448612 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.449854 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.449861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.450212 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.450806 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.450836 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.452143 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.452146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.453215 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.453225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.454175 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.454181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.454361 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.455049 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.455065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.457895 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":164,"threads_started":1}
W20260925 16:38:51.459599 9247 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:39025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.254:39025: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.465765 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.460022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:48971)
0925 16:38:51.460132 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.460147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.460221 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.460611 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.460614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.460740 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.460829 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.460833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.461965 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.461968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.462502 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.462509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.463154 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.463156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.463219 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.463750 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.463761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.465672 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:38:51.466005 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.460215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:48971 (local address 127.0.97.254:39025)
0925 16:38:51.460361 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.460368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.460385 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:38:51.460428 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.460432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.460491 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.460592 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.460945 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.461861 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.462622 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.463038 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.463165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.465879 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:48971: BlockingRecv error: recv got EOF from 127.0.97.193:48971 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.471510 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.466334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:46273)
0925 16:38:51.466569 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.466580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.466648 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.466889 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.466891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.467006 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.467071 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.467075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.468038 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.468041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.468604 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.468616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.469180 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.469182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.469238 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.469752 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.469763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.471420 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260925 16:38:51.471675 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.466461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:46273 (local address 127.0.97.254:39025)
0925 16:38:51.466577 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.466581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.466617 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:38:51.466727 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.466730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.466772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.466858 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.467168 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.467939 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.468737 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.469079 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.469175 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.471570 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:46273: BlockingRecv error: recv got EOF from 127.0.97.193:46273 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:38:51.471853 9247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:38:51.566953 9251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:51.567235 9251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:51.567313 9251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:51.571373 9251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:51.571460 9251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:51.571548 9251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:38:51.576661 9251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.194.keytab
--principal=oryx/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:40583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=33805
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:51.577785 9251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:51.578775 9251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:51.587989 9257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.587976 9256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.587980 9259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:51.588212 9251 server_base.cc:1034] running on GCE node
I20260925 16:38:51.588657 9251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:51.589429 9251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:51.590634 9251 hybrid_clock.cc:648] HybridClock initialized: now 1790354331590605 us; error 43 us; skew 500 ppm
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354331, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:51.593497 9251 init.cc:377] Logged in from keytab as oryx/127.0.97.194@KRBTEST.COM (short username oryx)
I20260925 16:38:51.594286 9251 webserver.cc:460] Webserver started at http://127.0.97.194:33805/ using document root <none> and password file <none>
I20260925 16:38:51.594839 9251 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:51.594916 9251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:51.598510 9251 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260925 16:38:51.600518 9266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.601744 9251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:38:51.601838 9251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "559693fe34844f75b6c6d7fb4a8b0e4b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "9fe4beb282d91b00895d3034713d3d2e"
server_key_iv: "5183d1a4412ceab26580d5b540ad2533"
server_key_version: "encryptionkey@0"
I20260925 16:38:51.602330 9251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:51.612268 9251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:51.615314 9251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:51.615450 9251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:51.615917 9251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:51.616639 9251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:51.616683 9251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.616719 9251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:51.616734 9251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.625456 9251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:40583
I20260925 16:38:51.625488 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:40583 every 8 connection(s)
I20260925 16:38:51.626386 9251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:38:51.631037 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9251
I20260925 16:38:51.631537 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:36795
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:39025
--webserver_port=45101
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:51.637879 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.628025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:34283 (local address 127.0.97.254:39025)
0925 16:38:51.628172 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.628177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.629038 (+ 861us) server_negotiation.cc:407] Connection header received
0925 16:38:51.630281 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.630284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.630332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.630406 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.631823 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.632335 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.633689 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.634029 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.634115 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.637750 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:34283: BlockingRecv error: recv got EOF from 127.0.97.194:34283 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:38:51.638437 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.628434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:34283)
0925 16:38:51.628884 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.628924 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.630092 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.630560 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.630566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.631017 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.631676 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.631701 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.632460 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.632467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.633561 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.633570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.634183 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.634189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.634417 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.635014 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.635038 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.637638 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
W20260925 16:38:51.638975 9380 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:39025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.254:39025: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.644881 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.639439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:59655)
0925 16:38:51.639547 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.639562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.639636 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.639905 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.639907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.640028 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.640107 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.640111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.641202 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.641204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.641833 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.641841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.642500 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.642503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.642570 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.643056 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.643068 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.644786 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:38:51.645043 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.639527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:59655 (local address 127.0.97.254:39025)
0925 16:38:51.639666 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.639672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.639689 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:38:51.639731 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.639737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.639790 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.639900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.640215 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.641093 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.641975 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.642367 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.642504 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.644932 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:59655: BlockingRecv error: recv got EOF from 127.0.97.194:59655 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.650595 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.645472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:34955)
0925 16:38:51.645621 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.645634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.645699 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.645939 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.645942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.646102 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.646184 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.646189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.646832 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.646835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.647683 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.647692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.648094 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.648098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.648180 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.648601 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.648629 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.650477 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 16:38:51.650717 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.645565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:34955 (local address 127.0.97.254:39025)
0925 16:38:51.645671 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.645674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.645685 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:38:51.645788 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.645790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.645838 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.645900 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.646284 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.646736 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.647808 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.647994 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.648065 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.650591 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:34955: BlockingRecv error: recv got EOF from 127.0.97.194:34955 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 16:38:51.650792 9380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:38:51.746354 9383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:38:51.746618 9383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:38:51.746661 9383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:38:51.750783 9383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:38:51.750849 9383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:38:51.750908 9383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:38:51.756594 9383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/krb5kdc/oryx_127.0.97.195.keytab
--principal=oryx/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:36795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=45101
--tserver_master_addrs=127.0.97.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:38:51.757830 9383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:38:51.758949 9383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:38:51.768702 9388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.768707 9389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:38:51.768707 9391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:38:51.769122 9383 server_base.cc:1034] running on GCE node
I20260925 16:38:51.769539 9383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:38:51.770206 9383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:38:51.771435 9383 hybrid_clock.cc:648] HybridClock initialized: now 1790354331771406 us; error 37 us; skew 500 ppm
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354331, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:38:51.774292 9383 init.cc:377] Logged in from keytab as oryx/127.0.97.195@KRBTEST.COM (short username oryx)
I20260925 16:38:51.775087 9383 webserver.cc:460] Webserver started at http://127.0.97.195:45101/ using document root <none> and password file <none>
I20260925 16:38:51.775646 9383 fs_manager.cc:359] Metadata directory not provided
I20260925 16:38:51.775717 9383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:38:51.779410 9383 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:38:51.781368 9398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.782583 9383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:38:51.782671 9383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "0385a32923174fc8800ec57f0146143b"
format_stamp: "Formatted at 2026-09-25 16:38:50 on dist-test-slave-kxl8"
server_key: "0791a3200cf1a8965c73d40878efc510"
server_key_iv: "b807772536a01d0c6243fec69b3dee29"
server_key_version: "encryptionkey@0"
I20260925 16:38:51.783155 9383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:38:51.801757 9383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:38:51.805011 9383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:38:51.805161 9383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:38:51.805763 9383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:38:51.806630 9383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:38:51.806675 9383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.806713 9383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:38:51.806730 9383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:38:51.815279 9383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:36795
I20260925 16:38:51.815343 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:36795 every 8 connection(s)
I20260925 16:38:51.816246 9383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:38:51.818065 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9383
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:51.828437 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.818054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:40797 (local address 127.0.97.254:39025)
0925 16:38:51.818189 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.818195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.819135 (+ 940us) server_negotiation.cc:407] Connection header received
0925 16:38:51.820882 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.820885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.820926 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.821004 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.822407 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.823049 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.824653 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.824960 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.825080 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.828323 (+ 3243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:40797: BlockingRecv error: recv got EOF from 127.0.97.195:40797 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:38:51.828639 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.819024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.0.97.254:39025)
0925 16:38:51.819353 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.819356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.819366 (+ 10us) server_negotiation.cc:407] Connection header received
0925 16:38:51.819401 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.819403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.819440 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.819510 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.820537 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.821202 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.822398 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.822723 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.822833 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.824412 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:38:51.824417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:38:51.824432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:38:51.824457 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:38:51.826301 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:51.826737 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:51.826743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:51.826748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:51.826892 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:38:51.827208 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:38:51.827214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:38:51.827217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:38:51.827659 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:38:51.827919 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 16:38:51.828109 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 16:38:51.828447 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":94,"threads_started":1}
I20260925 16:38:51.829138 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.818531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:40797)
0925 16:38:51.818990 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.819041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.820697 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.821157 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.821164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.821650 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.822281 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.822302 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.823188 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.823190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.824533 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.824541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.825108 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.825111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.825247 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.825818 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.825833 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.828221 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":139,"threads_started":1}
W20260925 16:38:51.829612 9512 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:39025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.254:39025: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.834915 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.830045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:54659)
0925 16:38:51.830196 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.830210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.830279 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.830612 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.830614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.830732 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.830855 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.830862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.831707 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.831709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.832232 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.832239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.832657 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.832659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.832721 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.833136 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.833147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.834810 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 16:38:51.835085 9516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.830133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:54659 (local address 127.0.97.254:39025)
0925 16:38:51.830241 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.830244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.830255 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:38:51.830382 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.830388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.830434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.830503 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.830991 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.831616 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.832357 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.832560 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.832642 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.834937 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:54659: BlockingRecv error: recv got EOF from 127.0.97.195:54659 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:51 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:51.839956 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:51.835423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:51841)
0925 16:38:51.835534 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:51.835550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:51.835625 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:51.835842 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:51.835845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:51.835984 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:51.836047 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.836051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.836744 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.836750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.837251 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:51.837258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.837754 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.837758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:51.837853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.838291 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:51.838305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:51.839829 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:38:51.840032 9516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:51.835513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:51841 (local address 127.0.97.254:39025)
0925 16:38:51.835633 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:51.835637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:51.835651 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:51.835692 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:51.835697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:51.835738 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:51.835823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:51.836153 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.836637 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.837446 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:51.837649 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:51.837754 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:51.839925 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:51841: BlockingRecv error: recv got EOF from 127.0.97.195:51841 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 16:38:51.840194 9512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:52.477968 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:52.472467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:45299)
0925 16:38:52.472733 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:52.472746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:52.472846 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:52.473113 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:52.473116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:52.473324 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:52.473438 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.473445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.474232 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.474235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.474823 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.474831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.475315 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.475319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.475418 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.475977 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:52.475989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:52.477846 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
W20260925 16:38:52.478127 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:52.472603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:45299 (local address 127.0.97.254:39025)
0925 16:38:52.472837 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:52.472841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:52.472854 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:52.472917 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:52.472919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:52.472960 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:52.473046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:52.473591 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.474125 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.474963 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.475191 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.475302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.477999 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:45299: BlockingRecv error: recv got EOF from 127.0.97.193:45299 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:38:52.658625 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:52.651637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:34949 (local address 127.0.97.254:39025)
0925 16:38:52.651745 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:52.651749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:52.651764 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:52.651807 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:52.651810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:52.651858 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:52.651962 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:52.652655 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.653603 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.654549 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.654853 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.654958 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.658478 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:34949: BlockingRecv error: recv got EOF from 127.0.97.194:34949 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 16:38:52.658684 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:52.651356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:34949)
0925 16:38:52.651598 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:52.651612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:52.651705 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:52.652121 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:52.652127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:52.652379 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:52.652532 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.652540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.653704 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.653710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.654423 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.654434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.654973 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.654977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.655069 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.655846 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:52.655862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:52.658293 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:52 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:52.846180 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:52.840813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:47875)
0925 16:38:52.841076 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:52.841089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:52.841184 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:52.841425 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:52.841427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:52.841596 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:52.841703 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.841708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.842413 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.842416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.842979 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:52.842987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.843380 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.843383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:52.843450 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.843918 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:52.843930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:52.846005 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
W20260925 16:38:52.846216 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:52.840905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:47875 (local address 127.0.97.254:39025)
0925 16:38:52.841021 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:52.841024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:52.841159 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:38:52.841265 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:52.841267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:52.841329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:52.841395 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:52.841800 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.842320 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.843082 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:52.843288 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:52.843363 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:52.846113 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:47875: BlockingRecv error: recv got EOF from 127.0.97.195:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:53.484230 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:53.478752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:36031)
0925 16:38:53.479028 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:53.479042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:53.479131 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:53.479385 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:53.479386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:53.479560 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:53.479651 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.479655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.480377 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.480379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.480951 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.480960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.481467 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.481470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.481536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.482026 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:53.482037 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:53.484072 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
W20260925 16:38:53.484350 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:53.478831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:36031 (local address 127.0.97.254:39025)
0925 16:38:53.479093 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:53.479096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:53.479109 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:38:53.479227 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:53.479229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:53.479269 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:53.479347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:53.479750 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.480274 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.481057 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.481268 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.481410 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.484210 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:36031: BlockingRecv error: recv got EOF from 127.0.97.193:36031 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:53.665060 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:53.659405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:33105)
0925 16:38:53.659667 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:53.659681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:53.659822 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:53.660089 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:53.660092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:53.660296 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:53.660383 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.660388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.661154 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.661159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.661804 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.661819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.662274 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.662278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.662353 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.662931 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:53.662942 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:53.664844 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
W20260925 16:38:53.665100 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:53.659463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:33105 (local address 127.0.97.254:39025)
0925 16:38:53.659628 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:53.659632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:53.659763 (+ 131us) server_negotiation.cc:407] Connection header received
0925 16:38:53.659926 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:53.659930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:53.659976 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:53.660066 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:53.660498 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.661043 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.661930 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.662166 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.662273 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.664987 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:33105: BlockingRecv error: recv got EOF from 127.0.97.194:33105 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:53 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:53.852602 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:53.846923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:48159)
0925 16:38:53.847255 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:53.847272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:53.847418 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:53.847684 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:53.847687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:53.847914 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:53.848007 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.848012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.848779 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.848786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.849423 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:53.849432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.849890 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.849894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:53.849984 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.850465 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:53.850477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:53.852403 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
W20260925 16:38:53.852619 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:53.847103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:48159 (local address 127.0.97.254:39025)
0925 16:38:53.847249 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:53.847254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:53.847353 (+ 99us) server_negotiation.cc:407] Connection header received
0925 16:38:53.847515 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:53.847519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:53.847573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:53.847663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:53.848120 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.848654 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.849539 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:53.849776 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:53.849888 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:53.852522 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:48159: BlockingRecv error: recv got EOF from 127.0.97.195:48159 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:54.490684 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:54.484999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:44703)
0925 16:38:54.485300 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:54.485314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:54.485408 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:54.485646 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:54.485648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:54.485822 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:54.485921 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.485926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.486772 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.486774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.487491 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.487500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.487962 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.487964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.488023 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.488547 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:54.488558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:54.490565 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":114,"threads_started":1}
W20260925 16:38:54.490839 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:54.485070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:44703 (local address 127.0.97.254:39025)
0925 16:38:54.485343 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:54.485348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:54.485363 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:38:54.485487 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:54.485489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:54.485535 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:54.485612 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:54.486038 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.486677 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.487608 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.487862 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.487947 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.490705 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:44703: BlockingRecv error: recv got EOF from 127.0.97.193:44703 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:54.671392 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:54.665972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:56213)
0925 16:38:54.666250 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:54.666263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:54.666378 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:54.666593 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:54.666594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:54.666767 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:54.666870 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.666875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.667564 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.667566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.668155 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.668163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.668625 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.668628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.668715 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.669359 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:54.669372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:54.671276 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
W20260925 16:38:54.671535 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:54.666077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:56213 (local address 127.0.97.254:39025)
0925 16:38:54.666268 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:54.666273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:54.666318 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:38:54.666449 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:54.666451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:54.666491 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:54.666559 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:54.666975 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.667477 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.668281 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.668509 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.668622 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.671416 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:56213: BlockingRecv error: recv got EOF from 127.0.97.194:56213 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:54 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:54.859138 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:54.853425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:37613)
0925 16:38:54.853763 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:54.853780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:54.853893 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:54.854190 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:54.854194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:54.854454 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:54.854569 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.854576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.855316 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.855320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.855892 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:54.855900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.856393 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.856428 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:54.856537 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.857037 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:54.857048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:54.859026 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
W20260925 16:38:54.859261 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:54.853503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:37613 (local address 127.0.97.254:39025)
0925 16:38:54.853762 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:54.853782 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:54.853832 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:38:54.853993 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:54.853995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:54.854065 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:54.854148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:54.854678 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.855188 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.855993 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:54.856293 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:54.856370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:54.859148 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:37613: BlockingRecv error: recv got EOF from 127.0.97.195:37613 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:55.496910 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:55.491639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:40395)
0925 16:38:55.491924 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:55.491938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:55.492020 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:55.492252 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:55.492254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:55.492429 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:55.492522 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.492526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.493446 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.493450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.494042 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.494051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.494495 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.494497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.494556 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.495045 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:55.495056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:55.496769 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
W20260925 16:38:55.497207 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:55.491680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:40395 (local address 127.0.97.254:39025)
0925 16:38:55.491958 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:55.491961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:55.491990 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:38:55.492105 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:55.492107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:55.492149 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:55.492216 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:55.492663 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.493229 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.494171 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.494396 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.494483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.497033 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:40395: BlockingRecv error: recv got EOF from 127.0.97.193:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:55.677827 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:55.672288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:41453)
0925 16:38:55.672556 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:55.672570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:55.672668 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:55.672952 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:55.672954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:55.673128 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:55.673218 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.673223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.674034 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.674037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.674588 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.674596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.675066 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.675069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.675130 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.675628 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:55.675639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:55.677639 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
W20260925 16:38:55.677861 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:55.672441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:41453 (local address 127.0.97.254:39025)
0925 16:38:55.672571 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:55.672574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:55.672639 (+ 65us) server_negotiation.cc:407] Connection header received
0925 16:38:55.672748 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:55.672750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:55.672833 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:55.672914 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:55.673379 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.673932 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.674700 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.674963 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.675049 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.677772 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:41453: BlockingRecv error: recv got EOF from 127.0.97.194:41453 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:55 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:55.865908 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:55.860259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:36299)
0925 16:38:55.860551 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:55.860567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:55.860692 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:55.860995 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:55.860998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:55.861204 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:55.861319 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.861326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.862083 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.862087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.862649 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:55.862657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.863104 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.863108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:55.863191 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.863676 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:55.863689 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:55.865795 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
W20260925 16:38:55.866154 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:55.860309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:36299 (local address 127.0.97.254:39025)
0925 16:38:55.860483 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:55.860487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:55.860650 (+ 163us) server_negotiation.cc:407] Connection header received
0925 16:38:55.860790 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:55.860793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:55.860859 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:55.860971 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:55.861455 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.861974 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.862766 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:55.862992 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:55.863099 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:55.865926 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:36299: BlockingRecv error: recv got EOF from 127.0.97.195:36299 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:56.508423 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:56.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:44033)
0925 16:38:56.498036 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:56.498054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:56.498171 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:56.498397 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:56.498399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:56.498573 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:56.498671 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.498675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.499473 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.499475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.500131 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.500140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.500573 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.500575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.500635 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.505805 (+ 5170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:56.505820 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:56.508247 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
W20260925 16:38:56.508636 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:56.497865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:44033 (local address 127.0.97.254:39025)
0925 16:38:56.498127 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:56.498131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:56.498145 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:38:56.498260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:56.498262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:56.498305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:56.498399 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:56.498763 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.499365 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.500242 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.500472 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.500549 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.508470 (+ 7921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:44033: BlockingRecv error: recv got EOF from 127.0.97.193:44033 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:56.684350 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:56.678714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:40171)
0925 16:38:56.679050 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:56.679066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:56.679199 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:56.679489 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:56.679492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:56.679705 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:56.679788 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.679793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.680541 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.680548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.681137 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.681145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.681675 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.681679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.681763 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.682362 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:56.682374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:56.684168 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260925 16:38:56.684420 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:56.678876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:40171 (local address 127.0.97.254:39025)
0925 16:38:56.679029 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:56.679033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:56.679151 (+ 118us) server_negotiation.cc:407] Connection header received
0925 16:38:56.679308 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:56.679312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:56.679374 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:56.679465 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:56.679911 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.680434 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.681259 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.681534 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.681675 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.684287 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:40171: BlockingRecv error: recv got EOF from 127.0.97.194:40171 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:56 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:56.871909 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:56.866774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:46839)
0925 16:38:56.867037 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:56.867050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:56.867137 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:56.867353 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:56.867355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:56.867526 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:56.867614 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.867618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.868324 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.868326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.868924 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:56.868932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.869376 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.869379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:56.869451 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.869945 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:56.869955 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:56.871722 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
W20260925 16:38:56.871944 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:56.866822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:46839 (local address 127.0.97.254:39025)
0925 16:38:56.866959 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:56.866964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:56.867123 (+ 159us) server_negotiation.cc:407] Connection header received
0925 16:38:56.867217 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:56.867219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:56.867262 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:56.867320 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:56.867705 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.868235 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.869024 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:56.869229 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:56.869324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:56.871840 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:46839: BlockingRecv error: recv got EOF from 127.0.97.195:46839 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:57.515719 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:57.509429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:34105)
0925 16:38:57.509721 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:57.509736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:57.509892 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:57.510154 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:57.510156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:57.510370 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:57.510480 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.510486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.511333 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.511337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.512153 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.512164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.512770 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.512774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.512879 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.513530 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:57.513541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:57.515594 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260925 16:38:57.515956 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:57.509441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34105 (local address 127.0.97.254:39025)
0925 16:38:57.509724 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:57.509728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:57.509815 (+ 87us) server_negotiation.cc:407] Connection header received
0925 16:38:57.509984 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:57.509986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:57.510031 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:57.510111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:57.510634 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.511209 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.512289 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.512619 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.512744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.515807 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:34105: BlockingRecv error: recv got EOF from 127.0.97.193:34105 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:57.691157 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:57.685342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:44511)
0925 16:38:57.685625 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:57.685638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:57.685739 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:57.685988 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:57.685990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:57.686173 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:57.686262 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.686266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.687186 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.687189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.687897 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.687907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.688491 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.688495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.688582 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.689079 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:57.689090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:57.690987 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":130,"threads_started":1}
W20260925 16:38:57.691200 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:57.685436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:44511 (local address 127.0.97.254:39025)
0925 16:38:57.685620 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:57.685625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:57.685714 (+ 89us) server_negotiation.cc:407] Connection header received
0925 16:38:57.685821 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:57.685824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:57.685865 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:57.685942 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:57.686450 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.687012 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.688047 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.688348 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.688477 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.691101 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:44511: BlockingRecv error: recv got EOF from 127.0.97.194:44511 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:57 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:57.878482 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:57.872747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:34337)
0925 16:38:57.873065 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:57.873082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:57.873197 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:57.873494 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:57.873497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:57.873741 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:57.873958 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.873969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.874716 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.874718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.875279 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:57.875286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.875747 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.875749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:57.875808 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.876332 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:57.876343 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:57.878312 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":154,"threads_started":1}
W20260925 16:38:57.878541 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:57.872806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:34337 (local address 127.0.97.254:39025)
0925 16:38:57.872922 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:57.872925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:57.873164 (+ 239us) server_negotiation.cc:407] Connection header received
0925 16:38:57.873304 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:57.873307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:57.873366 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:57.873456 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:57.874097 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.874623 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.875407 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:57.875644 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:57.875738 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:57.878438 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:34337: BlockingRecv error: recv got EOF from 127.0.97.195:34337 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:58.522351 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:58.516510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:42941)
0925 16:38:58.516826 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:58.516843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:58.516943 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:58.517210 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:58.517213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:58.517480 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:58.517591 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.517598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.518346 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.518348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.518886 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.518893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.519285 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.519288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.519361 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.519975 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:58.519992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:58.522180 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
W20260925 16:38:58.522449 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:58.516608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:42941 (local address 127.0.97.254:39025)
0925 16:38:58.516869 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:58.516873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:58.516890 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:38:58.517030 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:58.517033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:58.517073 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:58.517144 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:58.517703 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.518244 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.518982 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.519190 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.519263 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.522307 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:42941: BlockingRecv error: recv got EOF from 127.0.97.193:42941 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:58.697865 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:58.691887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:57171)
0925 16:38:58.692258 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:58.692271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:58.692409 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:58.692693 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:58.692696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:58.692911 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:58.692998 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.693003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.693838 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.693843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.694490 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.694499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.695096 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.695102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.695181 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.695727 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:58.695738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:58.697638 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":83,"threads_started":1}
W20260925 16:38:58.697926 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:58.692039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:57171 (local address 127.0.97.254:39025)
0925 16:38:58.692208 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:58.692213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:58.692347 (+ 134us) server_negotiation.cc:407] Connection header received
0925 16:38:58.692518 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:58.692521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:58.692577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:58.692669 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:58.693112 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.693725 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.694611 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.694931 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.695060 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.697777 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:57171: BlockingRecv error: recv got EOF from 127.0.97.194:57171 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:58 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:58.884724 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:58.879305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:51375)
0925 16:38:58.879645 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:58.879658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:58.879783 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:58.880084 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:58.880086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:58.880294 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:58.880383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.880389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.881129 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.881134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.881758 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:58.881767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.882229 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.882232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:58.882314 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.882801 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:58.882812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:58.884615 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":92,"threads_started":1}
W20260925 16:38:58.884833 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:58.879426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:51375 (local address 127.0.97.254:39025)
0925 16:38:58.879552 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:58.879557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:58.879734 (+ 177us) server_negotiation.cc:407] Connection header received
0925 16:38:58.879916 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:58.879921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:58.879977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:58.880064 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:58.880501 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.881024 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.881884 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:58.882119 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:58.882224 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:58.884746 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:51375: BlockingRecv error: recv got EOF from 127.0.97.195:51375 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:59.528915 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:59.523215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:56595)
0925 16:38:59.523501 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:59.523515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:59.523604 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:59.523912 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:59.523915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:59.524094 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:59.524204 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.524209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.524940 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.524943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.525538 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.525547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.525968 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.525971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.526034 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.526641 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:59.526653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:59.528721 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
W20260925 16:38:59.529268 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:59.523415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:56595 (local address 127.0.97.254:39025)
0925 16:38:59.523650 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:59.523654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:59.523666 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:38:59.523701 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:59.523703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:59.523743 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:59.523820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:59.524315 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.524836 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.525640 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.525860 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.525954 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.529003 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:56595: BlockingRecv error: recv got EOF from 127.0.97.193:56595 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:59.704502 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:59.698768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:39713)
0925 16:38:59.699060 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:59.699073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:59.699209 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:59.699480 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:59.699482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:59.699652 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:59.699744 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.699749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.700718 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.700720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.701367 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.701375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.701823 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.701826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.701887 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.702447 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:59.702458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:59.704316 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20260925 16:38:59.704538 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:59.698823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:39713 (local address 127.0.97.254:39025)
0925 16:38:59.699009 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:59.699014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:59.699175 (+ 161us) server_negotiation.cc:407] Connection header received
0925 16:38:59.699317 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:59.699319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:59.699366 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:59.699444 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:59.699873 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.700613 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.701502 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.701721 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.701811 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.704440 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:39713: BlockingRecv error: recv got EOF from 127.0.97.194:39713 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:38:59 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:38:59.891261 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:38:59.885634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:49355)
0925 16:38:59.885926 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:38:59.885940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:38:59.886040 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:38:59.886261 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:38:59.886264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:38:59.886461 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:38:59.886563 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.886568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.887349 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.887355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.887949 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:38:59.887958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.888395 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.888398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:38:59.888475 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.889034 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:38:59.889045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:38:59.891082 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
W20260925 16:38:59.891276 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:38:59.885692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:49355 (local address 127.0.97.254:39025)
0925 16:38:59.885838 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:38:59.885845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:38:59.886013 (+ 168us) server_negotiation.cc:407] Connection header received
0925 16:38:59.886119 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:38:59.886121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:38:59.886167 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:38:59.886233 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:38:59.886673 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.887242 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.888068 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:38:59.888285 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:38:59.888392 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:38:59.891185 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:49355: BlockingRecv error: recv got EOF from 127.0.97.195:49355 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:00.535342 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:00.529709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:43421)
0925 16:39:00.530042 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:00.530059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:00.530172 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:00.530487 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:00.530491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:00.530688 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:00.530794 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.530801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.531554 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.531557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.532156 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.532165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.532564 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.532567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.532626 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.533190 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:00.533203 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:00.535218 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
W20260925 16:39:00.535645 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:00.529759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:43421 (local address 127.0.97.254:39025)
0925 16:39:00.530024 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:00.530028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:00.530134 (+ 106us) server_negotiation.cc:407] Connection header received
0925 16:39:00.530275 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:00.530279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:00.530333 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:00.530443 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:00.530895 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.531431 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.532253 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.532476 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.532547 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.535379 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:43421: BlockingRecv error: recv got EOF from 127.0.97.193:43421 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:00.710603 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:00.705361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:45939)
0925 16:39:00.705628 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:00.705641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:00.705738 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:00.705967 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:00.705969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:00.706188 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:00.706316 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.706324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.707092 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.707094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.707637 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.707644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.708027 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.708029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.708089 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.708634 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:00.708647 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:00.710492 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
W20260925 16:39:00.710736 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:00.705414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:45939 (local address 127.0.97.254:39025)
0925 16:39:00.705556 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:00.705561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:00.705707 (+ 146us) server_negotiation.cc:407] Connection header received
0925 16:39:00.705815 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:00.705817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:00.705859 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:00.705928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:00.706435 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.707005 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.707730 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.707934 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.708007 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.710618 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:45939: BlockingRecv error: recv got EOF from 127.0.97.194:45939 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:00 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:00.897153 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:00.891970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:58559)
0925 16:39:00.892254 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:00.892267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:00.892360 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:00.892588 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:00.892590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:00.892760 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:00.892856 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.892860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.893666 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.893669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.894239 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:00.894247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.894663 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.894665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:00.894727 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.895300 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:00.895312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:00.897041 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
W20260925 16:39:00.897341 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:00.892064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:58559 (local address 127.0.97.254:39025)
0925 16:39:00.892192 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:00.892196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:00.892328 (+ 132us) server_negotiation.cc:407] Connection header received
0925 16:39:00.892445 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:00.892447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:00.892490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:00.892554 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:00.892989 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.893573 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.894335 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:00.894563 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:00.894639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:00.897166 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:58559: BlockingRecv error: recv got EOF from 127.0.97.195:58559 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:01.548199 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:01.536181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:37451)
0925 16:39:01.536487 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:01.536501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:01.536596 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:01.536877 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:01.536879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:01.537067 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:01.537162 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.537167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.537957 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.537960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.538530 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.538538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.538934 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.538936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.538995 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.546186 (+ 7191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:01.546198 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:01.548087 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":71,"threads_started":1}
W20260925 16:39:01.548394 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:01.536259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:37451 (local address 127.0.97.254:39025)
0925 16:39:01.536570 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:01.536573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:01.536586 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:39:01.536680 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:01.536682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:01.536723 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:01.536799 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:01.537258 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.537852 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.538632 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.538840 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.538914 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.548224 (+ 9310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:37451: BlockingRecv error: recv got EOF from 127.0.97.193:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:01.716567 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:01.711450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:47579)
0925 16:39:01.711715 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:01.711728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:01.711818 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:01.712060 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:01.712063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:01.712241 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:01.712333 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.712338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.713059 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.713061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.713635 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.713643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.714056 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.714059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.714124 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.714612 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:01.714629 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:01.716453 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
W20260925 16:39:01.716699 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:01.711543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:47579 (local address 127.0.97.254:39025)
0925 16:39:01.711680 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:01.711683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:01.711798 (+ 115us) server_negotiation.cc:407] Connection header received
0925 16:39:01.711884 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:01.711886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:01.711926 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:01.711993 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:01.712455 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.712967 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.713733 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.713953 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.714030 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.716581 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:47579: BlockingRecv error: recv got EOF from 127.0.97.194:47579 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:01 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:01.904001 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:01.898060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:33497)
0925 16:39:01.898372 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:01.898388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:01.898526 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:01.898793 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:01.898796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:01.899033 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:01.899124 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.899130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.899909 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.899914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.900474 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:01.900481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.900942 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.900946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:01.901031 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.901571 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:01.901582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:01.903794 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
W20260925 16:39:01.904012 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:01.898215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:33497 (local address 127.0.97.254:39025)
0925 16:39:01.898363 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:01.898375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:01.898472 (+ 97us) server_negotiation.cc:407] Connection header received
0925 16:39:01.898625 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:01.898629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:01.898685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:01.898770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:01.899245 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.899794 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.900592 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:01.900829 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:01.900944 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:01.903908 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:33497: BlockingRecv error: recv got EOF from 127.0.97.195:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:02.554430 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:02.548950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:60925)
0925 16:39:02.549261 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:02.549329 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:02.549443 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:02.549681 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:02.549683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:02.549922 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:02.550017 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.550022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.550759 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.550762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.551382 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.551391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.551794 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.551797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.551901 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.552396 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:02.552407 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:02.554254 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
W20260925 16:39:02.554502 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:02.549080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:60925 (local address 127.0.97.254:39025)
0925 16:39:02.549343 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:02.549348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:02.549400 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:39:02.549528 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:02.549530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:02.549573 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:02.549647 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:02.550115 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.550642 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.551486 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.551702 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.551778 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.554369 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:60925: BlockingRecv error: recv got EOF from 127.0.97.193:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:02.724956 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:02.717599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:60467)
0925 16:39:02.717917 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:02.717932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:02.718035 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:02.718379 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:02.718382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:02.718611 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:02.718754 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.718763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.719852 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.719872 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.720883 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.720897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.721524 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.721527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.721633 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.722225 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:02.722237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:02.724758 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":69,"threads_started":1}
W20260925 16:39:02.725116 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:02.717974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:60467 (local address 127.0.97.254:39025)
0925 16:39:02.718131 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:02.718136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:02.718150 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:39:02.718190 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:02.718194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:02.718251 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:02.718336 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:02.718852 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.719727 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.721023 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.721394 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.721505 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.724962 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:60467: BlockingRecv error: recv got EOF from 127.0.97.194:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:02 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:02.910233 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:02.904717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:49387)
0925 16:39:02.905020 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:02.905037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:02.905148 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:02.905488 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:02.905491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:02.905696 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:02.905783 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.905788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.906574 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.906579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.907142 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:02.907149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.907596 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.907599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:02.907686 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.908239 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:02.908260 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:02.910062 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260925 16:39:02.910291 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:02.904824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:49387 (local address 127.0.97.254:39025)
0925 16:39:02.905015 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:02.905019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:02.905094 (+ 75us) server_negotiation.cc:407] Connection header received
0925 16:39:02.905305 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:02.905311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:02.905372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:02.905470 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:02.905906 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.906460 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.907258 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:02.907486 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:02.907594 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:02.910187 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:49387: BlockingRecv error: recv got EOF from 127.0.97.195:49387 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:03.560794 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:03.555283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:42567)
0925 16:39:03.555628 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:03.555647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:03.555748 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:03.556047 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:03.556049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:03.556243 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:03.556336 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.556340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.557158 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.557161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.557802 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.557815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.558263 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.558265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.558325 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.558821 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:03.558832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:03.560676 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":71,"threads_started":1}
W20260925 16:39:03.561012 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:03.555327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:42567 (local address 127.0.97.254:39025)
0925 16:39:03.555646 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:03.555651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:03.555703 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:39:03.555874 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:03.555877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:03.555920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:03.556006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:03.556427 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.557062 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.557931 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.558167 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.558250 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.560810 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:42567: BlockingRecv error: recv got EOF from 127.0.97.193:42567 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:03.731634 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:03.725844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:43861)
0925 16:39:03.726139 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:03.726154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:03.726275 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:03.726624 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:03.726627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:03.726866 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:03.727080 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.727092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.727841 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.727843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.728394 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.728402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.729017 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.729020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.729084 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.729678 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:03.729692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:03.731526 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260925 16:39:03.731804 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:03.725874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:43861 (local address 127.0.97.254:39025)
0925 16:39:03.726073 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:03.726078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:03.726398 (+ 320us) server_negotiation.cc:407] Connection header received
0925 16:39:03.726442 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:03.726445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:03.726500 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:03.726588 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:03.727215 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.727749 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.728522 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.728742 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.728918 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.731652 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:43861: BlockingRecv error: recv got EOF from 127.0.97.194:43861 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:03 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:03.916584 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:03.911044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:45095)
0925 16:39:03.911473 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:03.911486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:03.911625 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:03.911947 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:03.911952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:03.912192 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:03.912289 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.912295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.913063 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.913066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.913658 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:03.913666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.914128 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.914132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:03.914197 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.914686 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:03.914696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:03.916467 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":105,"threads_started":1}
W20260925 16:39:03.916708 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:03.911175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:45095 (local address 127.0.97.254:39025)
0925 16:39:03.911308 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:03.911313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:03.911570 (+ 257us) server_negotiation.cc:407] Connection header received
0925 16:39:03.911735 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:03.911738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:03.911795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:03.911886 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:03.912398 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.912944 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.913764 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:03.913997 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:03.914081 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:03.916593 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:45095: BlockingRecv error: recv got EOF from 127.0.97.195:45095 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:04.567322 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:04.561636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:55049)
0925 16:39:04.561900 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:04.561914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:04.562020 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:04.562286 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:04.562288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:04.562470 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:04.562563 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.562568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.563330 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.563333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.563873 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.563881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.564419 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.564422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.564528 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.565200 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:04.565214 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:04.567199 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
W20260925 16:39:04.567510 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:04.561701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:55049 (local address 127.0.97.254:39025)
0925 16:39:04.561962 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:04.561966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:04.561979 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:39:04.562123 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:04.562126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:04.562170 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:04.562252 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:04.562673 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.563226 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.563991 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.564200 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.564278 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.567328 (+ 3050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:55049: BlockingRecv error: recv got EOF from 127.0.97.193:55049 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:04.737819 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:04.732472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:43883)
0925 16:39:04.732771 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:04.732784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:04.732881 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:04.733113 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:04.733115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:04.733362 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:04.733461 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.733467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.734153 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.734155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.734706 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.734713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.735106 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.735109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.735172 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.735770 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:04.735781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:04.737652 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":133,"threads_started":1}
W20260925 16:39:04.737882 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:04.732612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:43883 (local address 127.0.97.254:39025)
0925 16:39:04.732751 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:04.732754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:04.732851 (+ 97us) server_negotiation.cc:407] Connection header received
0925 16:39:04.732965 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:04.732967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:04.733008 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:04.733080 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:04.733559 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.734063 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.734805 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.735008 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.735081 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.737778 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:43883: BlockingRecv error: recv got EOF from 127.0.97.194:43883 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:04 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:04.923025 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:04.917408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:55713)
0925 16:39:04.917688 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:04.917704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:04.917842 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:04.918121 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:04.918124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:04.918336 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:04.918427 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.918432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.919189 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.919194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.919765 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:04.919773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.920227 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.920231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:04.920314 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.920824 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:04.920835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:04.922838 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
W20260925 16:39:04.923231 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:04.917494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:55713 (local address 127.0.97.254:39025)
0925 16:39:04.917646 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:04.917651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:04.917789 (+ 138us) server_negotiation.cc:407] Connection header received
0925 16:39:04.917951 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:04.917954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:04.918011 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:04.918101 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:04.918556 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.919079 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.919891 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:04.920117 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:04.920227 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:04.923119 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:55713: BlockingRecv error: recv got EOF from 127.0.97.195:55713 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:05.573662 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:05.568096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:34485)
0925 16:39:05.568420 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:05.568433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:05.568526 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:05.568758 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:05.568760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:05.568962 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:05.569056 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.569061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.569865 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.569868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.570423 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.570431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.570857 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.570860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.570934 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.571525 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:05.571536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:05.573526 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
W20260925 16:39:05.573781 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:05.568209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34485 (local address 127.0.97.254:39025)
0925 16:39:05.568493 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:05.568497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:05.568511 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:39:05.568608 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:05.568610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:05.568651 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:05.568719 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:05.569197 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.569762 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.570531 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.570757 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.570836 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.573653 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:34485: BlockingRecv error: recv got EOF from 127.0.97.193:34485 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":58,"threads_started":1}
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:05.744091 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:05.738558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:54579)
0925 16:39:05.738850 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:05.738863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:05.739000 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:05.739274 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:05.739277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:05.739495 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:05.739583 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.739589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.740348 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.740353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.740954 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.740963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.741473 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.741477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.741563 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.742128 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:05.742140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:05.743926 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
W20260925 16:39:05.744154 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:05.738621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:54579 (local address 127.0.97.254:39025)
0925 16:39:05.738786 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:05.738792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:05.738942 (+ 150us) server_negotiation.cc:407] Connection header received
0925 16:39:05.739110 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:05.739117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:05.739161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:05.739248 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:05.739701 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.740239 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.741078 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.741362 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.741471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.744050 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:54579: BlockingRecv error: recv got EOF from 127.0.97.194:54579 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:05 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.195@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:05.928851 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:05.923762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.195:59439)
0925 16:39:05.924055 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:05.924069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:05.924168 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:05.924397 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:05.924401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:05.924584 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:05.924681 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.924686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.925428 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.925431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.926028 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:05.926037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.926425 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.926427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:05.926487 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.926965 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:05.926977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:05.928725 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":73,"threads_started":1}
W20260925 16:39:05.928977 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:05.923863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:59439 (local address 127.0.97.254:39025)
0925 16:39:05.924052 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:05.924056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:05.924141 (+ 85us) server_negotiation.cc:407] Connection header received
0925 16:39:05.924248 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:05.924250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:05.924293 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:05.924361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:05.924780 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.925326 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.926126 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:05.926330 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:05.926404 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:05.928865 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.195:59439: BlockingRecv error: recv got EOF from 127.0.97.195:59439 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 16:39:06 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:06 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.193@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:06.587142 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:06.574587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.193:36599)
0925 16:39:06.574864 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:06.574877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:06.574965 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:06.575220 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:06.575221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:06.575396 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:06.575506 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:06.575512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.576238 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.576240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:06.576796 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:06.576803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.577203 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.577205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:06.577261 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:06.584740 (+ 7479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:06.584753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:06.586998 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
W20260925 16:39:06.587275 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:06.574708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:36599 (local address 127.0.97.254:39025)
0925 16:39:06.574938 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:06.574942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:06.574955 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:39:06.575064 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:06.575066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:06.575108 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:06.575183 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:06.575604 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.576135 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.576895 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.577111 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.577182 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:06.587140 (+ 9958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.193:36599: BlockingRecv error: recv got EOF from 127.0.97.193:36599 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":57,"threads_started":1}
Sep 25 16:39:06 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:39:06 dist-test-slave-kxl8 krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.194@KRBTEST.COM for oryx/127.0.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:39:06.750463 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:06.744974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:39025 (local address 127.0.97.194:54177)
0925 16:39:06.745283 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:06.745324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:06.745468 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:06.745741 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:06.745744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:06.745967 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:06.746054 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:06.746059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.746808 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.746814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:06.747369 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:06.747376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.747824 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.747827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:06.747908 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:06.748383 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:06.748395 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:06.750344 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.254:39025: Server oryx/127.0.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
W20260925 16:39:06.750636 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:06.745093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:54177 (local address 127.0.97.254:39025)
0925 16:39:06.745268 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:06.745298 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:06.745421 (+ 123us) server_negotiation.cc:407] Connection header received
0925 16:39:06.745576 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:06.745581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:06.745626 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:06.745721 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:06.746173 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.746702 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.747485 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:06.747718 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:06.747821 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:06.750481 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.194:54177: BlockingRecv error: recv got EOF from 127.0.97.194:54177 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:39:06.823601 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9099
I20260925 16:39:06.829737 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9251
I20260925 16:39:06.834937 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9383
I20260925 16:39:06.839849 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9029
2026-09-25T16:39:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18009 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T16:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:39:06Z Disabled control of system clock
I20260925 16:39:06.869962 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38281
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:39:06.978260 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:06.978533 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:06.978565 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:06.982474 9579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:39:06.982522 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:06.982537 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:06.982554 9579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:39:06.982569 9579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:39:06.987351 9579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:38281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:06.988366 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:06.989735 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:06.998624 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:06.998629 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:06.998667 9579 server_base.cc:1034] running on GCE node
W20260925 16:39:06.998632 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:06.999513 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:07.000648 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:07.001808 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790354347001792 us; error 34 us; skew 500 ppm
I20260925 16:39:07.003750 9579 webserver.cc:460] Webserver started at http://127.0.97.254:32955/ using document root <none> and password file <none>
I20260925 16:39:07.004464 9579 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:07.004535 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:07.004779 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:07.007171 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "0aa085c54a3748f3b5730d236629dddb"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "182f5ff178033215438179736199d3e7"
server_key_iv: "524f238b9bc60776eb430dbaeab5fb64"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.007774 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "0aa085c54a3748f3b5730d236629dddb"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "182f5ff178033215438179736199d3e7"
server_key_iv: "524f238b9bc60776eb430dbaeab5fb64"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.011718 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 16:39:07.014236 9593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.015530 9579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:07.015682 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "0aa085c54a3748f3b5730d236629dddb"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "182f5ff178033215438179736199d3e7"
server_key_iv: "524f238b9bc60776eb430dbaeab5fb64"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.015802 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:07.033710 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:07.034415 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:07.034624 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:07.041554 9579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:38281
I20260925 16:39:07.041565 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:38281 every 8 connection(s)
I20260925 16:39:07.042503 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:39:07.045153 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9579
I20260925 16:39:07.045262 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:39:07.045607 391 external_mini_cluster.cc:1428] Setting key 320575db5229183f69ab53594bb3f9cd
I20260925 16:39:07.045861 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:07.052460 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb: Bootstrap starting.
I20260925 16:39:07.054837 9646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:07.055068 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.047019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.97.254:38281)
0925 16:39:07.047624 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.047639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.047673 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:39:07.048505 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.048524 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.048924 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.049214 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.049642 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.050569 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.051778 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.052240 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.052354 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.053192 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:07.053197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:07.053208 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:07.053685 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:07.053737 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:07.053756 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:07.053831 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:07.053995 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 16:39:07.054277 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":139,"threads_started":1}
I20260925 16:39:07.055645 9646 log.cc:824] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:07.057475 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb: No bootstrap required, opened a new log
I20260925 16:39:07.060757 9646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } }
I20260925 16:39:07.060966 9646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:07.061000 9646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa085c54a3748f3b5730d236629dddb, State: Initialized, Role: FOLLOWER
I20260925 16:39:07.061476 9646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } }
I20260925 16:39:07.061617 9646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:39:07.061681 9646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:39:07.061790 9646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:07.062636 9646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } }
I20260925 16:39:07.062959 9646 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa085c54a3748f3b5730d236629dddb; no voters:
I20260925 16:39:07.063274 9646 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:39:07.063381 9651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:07.063625 9651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [term 1 LEADER]: Becoming Leader. State: Replica: 0aa085c54a3748f3b5730d236629dddb, State: Running, Role: LEADER
I20260925 16:39:07.063964 9651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } }
I20260925 16:39:07.064481 9646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:39:07.065547 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa085c54a3748f3b5730d236629dddb. Latest consensus state: current_term: 1 leader_uuid: "0aa085c54a3748f3b5730d236629dddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } } }
I20260925 16:39:07.065673 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:07.066046 9660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:39:07.066010 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aa085c54a3748f3b5730d236629dddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa085c54a3748f3b5730d236629dddb" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38281 } } }
I20260925 16:39:07.066695 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:07.069113 9660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:39:07.075227 9660 catalog_manager.cc:1347] Generated new cluster ID: 6863030c4ad4466f9ab38f9a342095e5
I20260925 16:39:07.075304 9660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:39:07.083318 9660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:39:07.084466 9660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:39:07.097733 9660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa085c54a3748f3b5730d236629dddb: Generated new TSK 0
I20260925 16:39:07.098610 9660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:39:07.111141 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--builtin_ntp_servers=127.0.97.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 16:39:07.221952 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:07.222250 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:07.222322 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:07.226420 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:07.226502 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:07.226593 9670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:39:07.231395 9670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:07.232492 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:07.233950 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:07.244187 9676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.244347 9670 server_base.cc:1034] running on GCE node
W20260925 16:39:07.244267 9675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.244179 9678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.244844 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:07.245570 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:07.246759 9670 hybrid_clock.cc:648] HybridClock initialized: now 1790354347246727 us; error 44 us; skew 500 ppm
I20260925 16:39:07.248492 9670 webserver.cc:460] Webserver started at http://127.0.97.193:34863/ using document root <none> and password file <none>
I20260925 16:39:07.249086 9670 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:07.249150 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:07.249435 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:07.251444 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "134a247d2e1a411c9b3537663f5d0948"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "8b256ddffb296238cb4a6b8fc04cd328"
server_key_iv: "06da708c94e7de5290e311c9ab702db3"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.252012 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "134a247d2e1a411c9b3537663f5d0948"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "8b256ddffb296238cb4a6b8fc04cd328"
server_key_iv: "06da708c94e7de5290e311c9ab702db3"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.255648 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:39:07.258177 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.259402 9670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:07.259549 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "134a247d2e1a411c9b3537663f5d0948"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "8b256ddffb296238cb4a6b8fc04cd328"
server_key_iv: "06da708c94e7de5290e311c9ab702db3"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.259670 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:07.275038 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:07.275740 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:07.275943 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:07.276533 9670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:07.277570 9670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:07.277642 9670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.277715 9670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:07.277758 9670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.286572 9670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:42207
I20260925 16:39:07.286595 9797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:42207 every 8 connection(s)
I20260925 16:39:07.287559 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:39:07.296464 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.289393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:39105 (local address 127.0.97.254:38281)
0925 16:39:07.289536 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.289542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.290634 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 16:39:07.291979 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.291983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.292052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.292132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.293065 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.293682 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.294762 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.294959 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.295044 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.296092 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:07.296095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:07.296097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:07.296107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:07.296120 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:07.296127 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:07.296181 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:07.296308 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 16:39:07.296353 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:39:07.297323 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.289848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38281 (local address 127.0.97.193:39105)
0925 16:39:07.290484 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:07.290524 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:07.291776 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:07.292281 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:07.292287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:07.292369 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:07.292910 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.292925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.293790 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.293792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.294662 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.294671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.295090 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.295092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.295225 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.295863 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:07.295865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:07.295902 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:07.295969 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:07.296196 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:07.296201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:07.296205 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:39:07.296294 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:39:07.296501 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260925 16:39:07.297714 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9670
I20260925 16:39:07.297830 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:39:07.298174 391 external_mini_cluster.cc:1428] Setting key a10f47f5d1034812e16041a5ea66f902
I20260925 16:39:07.298593 9798 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38281
I20260925 16:39:07.298900 9798 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:07.299412 9798 heartbeater.cc:511] Master 127.0.97.254:38281 requested a full tablet report, sending...
I20260925 16:39:07.299990 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--builtin_ntp_servers=127.0.97.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:39:07.300812 9610 ts_manager.cc:194] Registered new tserver with Master: 134a247d2e1a411c9b3537663f5d0948 (127.0.97.193:42207)
I20260925 16:39:07.301999 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.193:39105
W20260925 16:39:07.408388 9801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:07.408689 9801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:07.408761 9801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:07.412729 9801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:07.412813 9801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:07.412902 9801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:39:07.417632 9801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:07.418761 9801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:07.420346 9801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:07.429682 9807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.429701 9809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.429685 9806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.430044 9801 server_base.cc:1034] running on GCE node
I20260925 16:39:07.430424 9801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:07.431062 9801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:07.432242 9801 hybrid_clock.cc:648] HybridClock initialized: now 1790354347432233 us; error 29 us; skew 500 ppm
I20260925 16:39:07.433979 9801 webserver.cc:460] Webserver started at http://127.0.97.194:40575/ using document root <none> and password file <none>
I20260925 16:39:07.434544 9801 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:07.434634 9801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:07.434861 9801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:07.436843 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "8d834fdd2d8e4eaf91db8a021cc2fdf6"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "7132e4f96b41510e965fbca42245f1cb"
server_key_iv: "a0f01da9bdc02e85968aab5a896f59a5"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.437470 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "8d834fdd2d8e4eaf91db8a021cc2fdf6"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "7132e4f96b41510e965fbca42245f1cb"
server_key_iv: "a0f01da9bdc02e85968aab5a896f59a5"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.441004 9801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:39:07.443468 9815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.444624 9801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:39:07.444756 9801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "8d834fdd2d8e4eaf91db8a021cc2fdf6"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "7132e4f96b41510e965fbca42245f1cb"
server_key_iv: "a0f01da9bdc02e85968aab5a896f59a5"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.444869 9801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:07.468434 9801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:07.469183 9801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:07.469408 9801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:07.470005 9801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:07.470973 9801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:07.471057 9801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.471158 9801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:07.471212 9801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.480320 9801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:45383
I20260925 16:39:07.480336 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:45383 every 8 connection(s)
I20260925 16:39:07.481240 9801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:39:07.486241 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9801
I20260925 16:39:07.486335 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:39:07.486635 391 external_mini_cluster.cc:1428] Setting key 5b18ced3416b7b24bc75968e086fdbe1
I20260925 16:39:07.488615 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--builtin_ntp_servers=127.0.97.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:39:07.491518 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.483059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:46097 (local address 127.0.97.254:38281)
0925 16:39:07.483190 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.483194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.484250 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 16:39:07.485801 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.485804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.485860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.485974 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.486999 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.487631 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.489327 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.489566 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.489663 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.490934 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:07.490940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:07.490945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:07.490958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:07.490985 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:07.490999 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:07.491083 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:07.491226 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 16:39:07.491297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:39:07.492359 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.483474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38281 (local address 127.0.97.194:46097)
0925 16:39:07.484105 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:07.484144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:07.485577 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:07.486159 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:07.486166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:07.486252 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:07.486859 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.486874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.487765 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.487771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.489152 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.489167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.489741 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.489748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.489986 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.490672 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:07.490673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:07.490713 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:07.490778 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:07.491089 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:07.491095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:07.491100 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:39:07.491199 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:39:07.491422 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260925 16:39:07.493579 9929 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38281
I20260925 16:39:07.493808 9929 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:07.494326 9929 heartbeater.cc:511] Master 127.0.97.254:38281 requested a full tablet report, sending...
I20260925 16:39:07.495355 9610 ts_manager.cc:194] Registered new tserver with Master: 8d834fdd2d8e4eaf91db8a021cc2fdf6 (127.0.97.194:45383)
I20260925 16:39:07.495918 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.194:46097
W20260925 16:39:07.599922 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:07.600170 9932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:07.600206 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:07.604234 9932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:07.604283 9932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:07.604360 9932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:39:07.609041 9932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:07.610090 9932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:07.611420 9932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:07.620618 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.620667 9937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.620699 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.620926 9932 server_base.cc:1034] running on GCE node
I20260925 16:39:07.621316 9932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:07.621942 9932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:07.623116 9932 hybrid_clock.cc:648] HybridClock initialized: now 1790354347623106 us; error 35 us; skew 500 ppm
I20260925 16:39:07.624840 9932 webserver.cc:460] Webserver started at http://127.0.97.195:41929/ using document root <none> and password file <none>
I20260925 16:39:07.625500 9932 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:07.625586 9932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:07.625819 9932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:07.627769 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "4c16a266d2ef4f278509c3d4e9633c20"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "d1aeb7481f5e1ef38e32b34008aaafb1"
server_key_iv: "5cd9f5849b6c46a17e3c25c397c03733"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.628306 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "4c16a266d2ef4f278509c3d4e9633c20"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "d1aeb7481f5e1ef38e32b34008aaafb1"
server_key_iv: "5cd9f5849b6c46a17e3c25c397c03733"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.631765 9932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:39:07.634239 9946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.635423 9932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:07.635543 9932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "4c16a266d2ef4f278509c3d4e9633c20"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "d1aeb7481f5e1ef38e32b34008aaafb1"
server_key_iv: "5cd9f5849b6c46a17e3c25c397c03733"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.635646 9932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:07.666132 9932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:07.666908 9932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:07.667095 9932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:07.667691 9932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:07.668613 9932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:07.668679 9932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.668753 9932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:07.668794 9932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.677766 9932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:33235
I20260925 16:39:07.677790 10059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:33235 every 8 connection(s)
I20260925 16:39:07.678653 9932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:39:07.685165 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 9932
I20260925 16:39:07.685261 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:39:07.685591 391 external_mini_cluster.cc:1428] Setting key fb849d62357434d9a418996a2280859b
I20260925 16:39:07.688493 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.680420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:60449 (local address 127.0.97.254:38281)
0925 16:39:07.680585 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.680590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.681982 (+ 1392us) server_negotiation.cc:407] Connection header received
0925 16:39:07.683391 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.683395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.683452 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.683532 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.684388 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.684957 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.686536 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.686774 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.686889 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.688093 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:07.688095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:07.688097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:07.688108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:07.688127 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:07.688134 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:07.688193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:07.688341 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:39:07.688388 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:39:07.689437 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.680821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38281 (local address 127.0.97.195:60449)
0925 16:39:07.681838 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:07.681877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:07.683185 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:07.683696 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:07.683702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:07.683785 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:07.684250 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.684261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.685118 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.685121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.686387 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:07.686399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:07.686922 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:07.686926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:07.687117 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:07.687864 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:07.687865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:07.687905 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:07.687964 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:07.688218 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:07.688225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:07.688229 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:39:07.688327 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 16:39:07.688575 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":99,"threads_started":1}
I20260925 16:39:07.690467 10060 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38281
I20260925 16:39:07.690699 10060 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:07.691185 10060 heartbeater.cc:511] Master 127.0.97.254:38281 requested a full tablet report, sending...
I20260925 16:39:07.692108 9610 ts_manager.cc:194] Registered new tserver with Master: 4c16a266d2ef4f278509c3d4e9633c20 (127.0.97.195:33235)
I20260925 16:39:07.692659 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.195:60449
I20260925 16:39:07.699600 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:39:07.702606 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:07.701614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.97.254:38281)
0925 16:39:07.701761 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.701766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.701961 (+ 195us) server_negotiation.cc:407] Connection header received
0925 16:39:07.702089 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.702091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.702131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.702196 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.702448 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60374: BlockingRecv error: recv got EOF from 127.0.0.1:60374 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:39:07.702708 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9670
I20260925 16:39:07.708220 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9801
I20260925 16:39:07.713632 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9932
I20260925 16:39:07.718734 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 9579
2026-09-25T16:39:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (877 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T16:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:39:07Z Disabled control of system clock
I20260925 16:39:07.746851 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:40449
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:39:07.853930 10071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:07.854216 10071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:07.854305 10071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:07.858311 10071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:39:07.858388 10071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:07.858417 10071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:07.858455 10071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:39:07.858494 10071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:39:07.863214 10071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:40449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:07.864229 10071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:07.865579 10071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:07.873767 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:07.873852 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.873799 10071 server_base.cc:1034] running on GCE node
W20260925 16:39:07.873970 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:07.874436 10071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:07.875569 10071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:07.876749 10071 hybrid_clock.cc:648] HybridClock initialized: now 1790354347876723 us; error 40 us; skew 500 ppm
I20260925 16:39:07.878587 10071 webserver.cc:460] Webserver started at http://127.0.97.254:44815/ using document root <none> and password file <none>
I20260925 16:39:07.879166 10071 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:07.879261 10071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:07.879485 10071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:07.881599 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "a679301d59824174a56b64af122bf00e"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "c5531179945e2f225be0ffb9e7a75303"
server_key_iv: "47e7424c1138e3119894971e8df329fb"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.882167 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "a679301d59824174a56b64af122bf00e"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "c5531179945e2f225be0ffb9e7a75303"
server_key_iv: "47e7424c1138e3119894971e8df329fb"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.885751 10071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 16:39:07.888202 10085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:07.889510 10071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:07.889649 10071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "a679301d59824174a56b64af122bf00e"
format_stamp: "Formatted at 2026-09-25 16:39:07 on dist-test-slave-kxl8"
server_key: "c5531179945e2f225be0ffb9e7a75303"
server_key_iv: "47e7424c1138e3119894971e8df329fb"
server_key_version: "encryptionkey@0"
I20260925 16:39:07.889760 10071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:07.894258 10071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:07.894840 10071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:07.895022 10071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:07.901932 10071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:40449
I20260925 16:39:07.902011 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:40449 every 8 connection(s)
I20260925 16:39:07.902879 10071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:39:07.906103 10138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:07.911612 10138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e: Bootstrap starting.
I20260925 16:39:07.912012 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10071
I20260925 16:39:07.912129 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:39:07.912421 391 external_mini_cluster.cc:1428] Setting key ef793b53be74050871cad593cd8d7929
I20260925 16:39:07.914507 10138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:07.915248 10138 log.cc:824] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:07.917527 10138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e: No bootstrap required, opened a new log
I20260925 16:39:07.918059 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:07.913802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.97.254:40449)
0925 16:39:07.914446 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:07.914461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:07.914494 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:39:07.915146 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:07.915162 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:07.915544 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:07.915910 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:07.916699 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:07.916703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:07.916715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:07.916724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:07.916772 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:07.916787 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:07.916874 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:07.917021 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 16:39:07.917326 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":137,"threads_started":1}
I20260925 16:39:07.921655 10138 raft_consensus.cc:348] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } }
I20260925 16:39:07.921944 10138 raft_consensus.cc:374] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:07.922050 10138 raft_consensus.cc:729] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a679301d59824174a56b64af122bf00e, State: Initialized, Role: FOLLOWER
I20260925 16:39:07.922535 10138 consensus_queue.cc:260] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } }
I20260925 16:39:07.922699 10138 raft_consensus.cc:388] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:39:07.922770 10138 raft_consensus.cc:482] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:39:07.922895 10138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:07.923805 10138 raft_consensus.cc:504] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } }
I20260925 16:39:07.924160 10138 leader_election.cc:302] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a679301d59824174a56b64af122bf00e; no voters:
I20260925 16:39:07.924461 10138 leader_election.cc:288] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:39:07.924697 10143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:07.924942 10143 raft_consensus.cc:686] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [term 1 LEADER]: Becoming Leader. State: Replica: a679301d59824174a56b64af122bf00e, State: Running, Role: LEADER
I20260925 16:39:07.925225 10143 consensus_queue.cc:237] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } }
I20260925 16:39:07.926075 10138 sys_catalog.cc:564] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:39:07.927109 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a679301d59824174a56b64af122bf00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } } }
I20260925 16:39:07.927143 10145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a679301d59824174a56b64af122bf00e. Latest consensus state: current_term: 1 leader_uuid: "a679301d59824174a56b64af122bf00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a679301d59824174a56b64af122bf00e" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40449 } } }
I20260925 16:39:07.927264 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:07.927297 10145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:07.927660 10152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:39:07.931267 10152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:39:07.936698 10152 catalog_manager.cc:1347] Generated new cluster ID: 6330860700324f1a86b283f91c6b4337
I20260925 16:39:07.936764 10152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:39:07.944357 10152 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:39:07.944650 10152 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:39:07.957641 10152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a679301d59824174a56b64af122bf00e: Generated new TSK 0
I20260925 16:39:07.958335 10152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:39:07.969921 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--builtin_ntp_servers=127.0.97.212:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:39:08.083410 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:08.083683 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:08.083724 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:08.087950 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:08.088012 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:08.088109 10162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:39:08.092869 10162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:08.094066 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:08.095432 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:08.104626 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:08.104617 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:08.104768 10162 server_base.cc:1034] running on GCE node
W20260925 16:39:08.104839 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:08.105322 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:08.105943 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:08.107127 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790354348107114 us; error 26 us; skew 500 ppm
I20260925 16:39:08.108850 10162 webserver.cc:460] Webserver started at http://127.0.97.193:41317/ using document root <none> and password file <none>
I20260925 16:39:08.109485 10162 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:08.109581 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:08.109812 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:08.111804 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "432d8850ceaf4a5abd72ab2378cb4f28"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "d3e86d6a45eb6f60895578ddb24088a0"
server_key_iv: "b61aeeacf4734f50b925cf3c2321c4b7"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.112365 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "432d8850ceaf4a5abd72ab2378cb4f28"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "d3e86d6a45eb6f60895578ddb24088a0"
server_key_iv: "b61aeeacf4734f50b925cf3c2321c4b7"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.115888 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:39:08.118197 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.119382 10162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:08.119520 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "432d8850ceaf4a5abd72ab2378cb4f28"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "d3e86d6a45eb6f60895578ddb24088a0"
server_key_iv: "b61aeeacf4734f50b925cf3c2321c4b7"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.119628 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:08.124073 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:08.124641 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:08.124816 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:08.125607 10162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:08.126613 10162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:08.126681 10162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.126744 10162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:08.126797 10162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.135301 10162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:44803
I20260925 16:39:08.135339 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:44803 every 8 connection(s)
I20260925 16:39:08.136144 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:39:08.136328 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10162
I20260925 16:39:08.136443 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:39:08.136781 391 external_mini_cluster.cc:1428] Setting key f9c247406fc1454aa37f52f7986aa28a
I20260925 16:39:08.138809 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--builtin_ntp_servers=127.0.97.212:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:39:08.143146 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.137978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:43091 (local address 127.0.97.254:40449)
0925 16:39:08.138101 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:08.138106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:08.139368 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 16:39:08.141040 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:08.141044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:08.141096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:08.141181 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:08.142643 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:08.142649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:08.142653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:08.142669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:08.142699 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:08.142707 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:08.142790 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:08.143006 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 16:39:08.143045 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 16:39:08.144405 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.138447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40449 (local address 127.0.97.193:43091)
0925 16:39:08.139185 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:08.139241 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:08.140840 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:08.141380 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:08.141392 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:08.141602 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:08.142301 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:08.142305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:08.142362 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:08.142450 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:08.142807 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:08.142819 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:08.142826 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:39:08.143020 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 16:39:08.143362 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260925 16:39:08.145663 10290 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40449
I20260925 16:39:08.145910 10290 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:08.146396 10290 heartbeater.cc:511] Master 127.0.97.254:40449 requested a full tablet report, sending...
I20260925 16:39:08.147938 10102 ts_manager.cc:194] Registered new tserver with Master: 432d8850ceaf4a5abd72ab2378cb4f28 (127.0.97.193:44803)
W20260925 16:39:08.252543 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:08.252789 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:08.252853 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:08.257153 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:08.257212 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:08.257347 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:39:08.262109 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:08.263155 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:08.264510 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:08.273916 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:08.273908 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:08.273916 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:08.274124 10293 server_base.cc:1034] running on GCE node
I20260925 16:39:08.274642 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:08.275276 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:08.276476 10293 hybrid_clock.cc:648] HybridClock initialized: now 1790354348276454 us; error 29 us; skew 500 ppm
I20260925 16:39:08.278460 10293 webserver.cc:460] Webserver started at http://127.0.97.194:45117/ using document root <none> and password file <none>
I20260925 16:39:08.279052 10293 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:08.279140 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:08.279371 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:08.281440 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "cf67535624104753bed95c7da5525958"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "1403f7af8a946f195c6b32903fff1f7f"
server_key_iv: "404b6bc4959f964264cbb3234dfe6573"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.281994 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "cf67535624104753bed95c7da5525958"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "1403f7af8a946f195c6b32903fff1f7f"
server_key_iv: "404b6bc4959f964264cbb3234dfe6573"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.285669 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:39:08.288156 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.289413 10293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:08.289535 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "cf67535624104753bed95c7da5525958"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "1403f7af8a946f195c6b32903fff1f7f"
server_key_iv: "404b6bc4959f964264cbb3234dfe6573"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.289644 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:08.294340 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:08.294905 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:08.295083 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:08.295594 10293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:08.296422 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:08.296502 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.296590 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:08.296640 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.305104 10293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:43449
I20260925 16:39:08.305125 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:43449 every 8 connection(s)
I20260925 16:39:08.306017 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:39:08.307085 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10293
I20260925 16:39:08.307210 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:39:08.307516 391 external_mini_cluster.cc:1428] Setting key 3e29dd85a0be4533764118ba15d53555
I20260925 16:39:08.309775 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--builtin_ntp_servers=127.0.97.212:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:39:08.313290 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.307893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:58867 (local address 127.0.97.254:40449)
0925 16:39:08.308051 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:08.308055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:08.309422 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 16:39:08.311303 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:08.311308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:08.311357 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:08.311445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:08.312780 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:08.312783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:08.312786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:08.312797 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:08.312816 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:08.312825 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:08.312926 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:08.313101 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 16:39:08.313145 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:39:08.314126 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.308404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40449 (local address 127.0.97.194:58867)
0925 16:39:08.309256 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:08.309337 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:08.311050 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:08.311824 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:08.311831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:08.311917 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:08.312538 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:08.312540 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:08.312578 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:08.312643 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:08.312913 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:08.312921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:08.312926 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:39:08.313132 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 16:39:08.313381 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":130,"threads_started":1}
I20260925 16:39:08.315326 10421 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40449
I20260925 16:39:08.315608 10421 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:08.316097 10421 heartbeater.cc:511] Master 127.0.97.254:40449 requested a full tablet report, sending...
I20260925 16:39:08.317229 10102 ts_manager.cc:194] Registered new tserver with Master: cf67535624104753bed95c7da5525958 (127.0.97.194:43449)
W20260925 16:39:08.420145 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:08.420428 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:08.420502 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:08.424633 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:08.424726 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:08.424810 10424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:39:08.429894 10424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:08.430927 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:08.432371 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:08.442001 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:08.442009 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:08.442018 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:08.442242 10424 server_base.cc:1034] running on GCE node
I20260925 16:39:08.442601 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:08.443209 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:08.444391 10424 hybrid_clock.cc:648] HybridClock initialized: now 1790354348444301 us; error 103 us; skew 500 ppm
I20260925 16:39:08.446264 10424 webserver.cc:460] Webserver started at http://127.0.97.195:44877/ using document root <none> and password file <none>
I20260925 16:39:08.446833 10424 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:08.446928 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:08.447162 10424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:08.449187 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "756cf6f6234a40c8807485e3cecc9f49"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "c88adaa87c26e35c98e699451552590e"
server_key_iv: "c9db7f0e0673d9a9fac3483d8c9026ff"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.449815 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "756cf6f6234a40c8807485e3cecc9f49"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "c88adaa87c26e35c98e699451552590e"
server_key_iv: "c9db7f0e0673d9a9fac3483d8c9026ff"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.453397 10424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:39:08.455785 10438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.456939 10424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:39:08.457094 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "756cf6f6234a40c8807485e3cecc9f49"
format_stamp: "Formatted at 2026-09-25 16:39:08 on dist-test-slave-kxl8"
server_key: "c88adaa87c26e35c98e699451552590e"
server_key_iv: "c9db7f0e0673d9a9fac3483d8c9026ff"
server_key_version: "encryptionkey@0"
I20260925 16:39:08.457209 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:08.461862 10424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:08.462427 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:08.462564 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:08.463042 10424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:08.463893 10424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:08.463939 10424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.463977 10424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:08.463994 10424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:08.472919 10424 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:41933
I20260925 16:39:08.472939 10551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:41933 every 8 connection(s)
I20260925 16:39:08.473950 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:39:08.476151 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10424
I20260925 16:39:08.476238 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:39:08.476508 391 external_mini_cluster.cc:1428] Setting key e2a0f082560cc976b2ccb36f3f787324
I20260925 16:39:08.480212 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.475773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:46805 (local address 127.0.97.254:40449)
0925 16:39:08.475924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:08.475930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:08.477058 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 16:39:08.478606 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:08.478609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:08.478650 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:08.478732 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:08.479869 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:08.479871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:08.479873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:39:08.479883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:08.479901 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:39:08.479909 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:08.479977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:08.480074 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 16:39:08.480110 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:39:08.481053 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:08.476162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40449 (local address 127.0.97.195:46805)
0925 16:39:08.476908 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:08.476961 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:08.478393 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:08.478897 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:08.478904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:08.478993 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:08.479626 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:39:08.479628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:08.479668 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:39:08.479737 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:08.479967 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:08.479971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:08.479974 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:39:08.480099 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 16:39:08.480288 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260925 16:39:08.482137 10552 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40449
I20260925 16:39:08.482326 10552 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:08.482686 10552 heartbeater.cc:511] Master 127.0.97.254:40449 requested a full tablet report, sending...
I20260925 16:39:08.483579 10102 ts_manager.cc:194] Registered new tserver with Master: 756cf6f6234a40c8807485e3cecc9f49 (127.0.97.195:41933)
I20260925 16:39:08.490423 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:39:08.493412 10141 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:39:08.492304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.0.97.254:40449)
0925 16:39:08.492454 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:08.492459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:08.492703 (+ 244us) server_negotiation.cc:407] Connection header received
0925 16:39:08.492877 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:08.492882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:08.492924 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:08.493022 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:08.493244 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60906: BlockingRecv error: recv got EOF from 127.0.0.1:60906 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:39:08.493551 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10162
I20260925 16:39:08.499202 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10293
I20260925 16:39:08.504745 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10424
I20260925 16:39:08.511659 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10071
2026-09-25T16:39:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (793 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:39:08 dist-test-slave-kxl8 krb5kdc[10562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:39:08 dist-test-slave-kxl8 krb5kdc[10562](info): set up 2 sockets
Sep 25 16:39:08 dist-test-slave-kxl8 krb5kdc[10562](info): commencing operation
krb5kdc: starting...
W20260925 16:39:09.550426 391 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:39:09 dist-test-slave-kxl8 krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:39:09.671638 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:41531
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:09.780153 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:09.780476 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:09.780560 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:09.784570 10575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:39:09.784674 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:09.784714 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:09.784766 10575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:39:09.784806 10575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:39:09.790004 10575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:41531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:09.791162 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:09.792176 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:09.800802 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:09.800802 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:09.800895 10575 server_base.cc:1034] running on GCE node
W20260925 16:39:09.800796 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:09.801723 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:09.802829 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:09.804014 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790354349803985 us; error 42 us; skew 500 ppm
Sep 25 16:39:09 dist-test-slave-kxl8 krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:09.806753 10575 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:39:09.807706 10575 webserver.cc:460] Webserver started at http://127.0.97.254:38237/ using document root <none> and password file <none>
I20260925 16:39:09.808264 10575 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:09.808352 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:09.808578 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:09.810664 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d1286c922e4047578c86186e5da69764"
format_stamp: "Formatted at 2026-09-25 16:39:09 on dist-test-slave-kxl8"
server_key: "009a9b038956f3744214a473d65a9785"
server_key_iv: "6d3bcad078e8fd3082e10204d68f9640"
server_key_version: "encryptionkey@0"
I20260925 16:39:09.811225 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d1286c922e4047578c86186e5da69764"
format_stamp: "Formatted at 2026-09-25 16:39:09 on dist-test-slave-kxl8"
server_key: "009a9b038956f3744214a473d65a9785"
server_key_iv: "6d3bcad078e8fd3082e10204d68f9640"
server_key_version: "encryptionkey@0"
I20260925 16:39:09.814877 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:39:09.817229 10590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:09.818503 10575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:09.818643 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d1286c922e4047578c86186e5da69764"
format_stamp: "Formatted at 2026-09-25 16:39:09 on dist-test-slave-kxl8"
server_key: "009a9b038956f3744214a473d65a9785"
server_key_iv: "6d3bcad078e8fd3082e10204d68f9640"
server_key_version: "encryptionkey@0"
I20260925 16:39:09.818753 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:09.855495 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:09.858373 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:09.858587 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:09.865536 10575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:41531
I20260925 16:39:09.865824 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:41531 every 8 connection(s)
I20260925 16:39:09.866595 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:39:09.867944 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10575
I20260925 16:39:09.868039 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:39:09.868342 391 external_mini_cluster.cc:1428] Setting key 2ab0b129a37cd95e683e8e59fc70bdaf
I20260925 16:39:09.869647 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:09.876459 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: Bootstrap starting.
Sep 25 16:39:09 dist-test-slave-kxl8 krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:39:09.879148 10643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:09.880084 10643 log.cc:824] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:09.882200 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: No bootstrap required, opened a new log
I20260925 16:39:09.883440 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:09.869813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.97.254:41531)
0925 16:39:09.870218 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:09.870237 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:09.870279 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:39:09.871119 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:09.871141 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:09.871585 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:09.871956 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:09.872618 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:09.874146 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:09.874981 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:09.875698 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:09.875852 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:09.878503 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:09.878509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:09.878524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:09.878564 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:09.880512 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:09.880988 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:09.880994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:09.881000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:09.881119 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:09.881486 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:09.881489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:09.881491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:09.881898 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:09.882122 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:09.882409 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 16:39:09.882707 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260925 16:39:09.886063 10643 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } }
I20260925 16:39:09.886322 10643 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:09.886405 10643 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1286c922e4047578c86186e5da69764, State: Initialized, Role: FOLLOWER
I20260925 16:39:09.886865 10643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } }
I20260925 16:39:09.887027 10643 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:39:09.887102 10643 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:39:09.887223 10643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:09.888259 10643 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } }
I20260925 16:39:09.888616 10643 leader_election.cc:302] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1286c922e4047578c86186e5da69764; no voters:
I20260925 16:39:09.889003 10643 leader_election.cc:288] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:39:09.889132 10648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:09.889547 10648 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [term 1 LEADER]: Becoming Leader. State: Replica: d1286c922e4047578c86186e5da69764, State: Running, Role: LEADER
I20260925 16:39:09.889891 10648 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } }
I20260925 16:39:09.890380 10643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:39:09.891737 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1286c922e4047578c86186e5da69764. Latest consensus state: current_term: 1 leader_uuid: "d1286c922e4047578c86186e5da69764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } } }
I20260925 16:39:09.891903 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:09.891737 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1286c922e4047578c86186e5da69764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1286c922e4047578c86186e5da69764" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 41531 } } }
I20260925 16:39:09.892115 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764 [sys.catalog]: This master's current role is: LEADER
W20260925 16:39:09.895183 10663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:39:09.895277 10663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:39:09.895349 10657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:39:09.896049 10657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:39:09.902146 10657 catalog_manager.cc:1347] Generated new cluster ID: 474f51d05e7f4b6d9870ed560bae0744
I20260925 16:39:09.902230 10657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:39:09.929824 10657 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:39:09.931099 10657 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:39:09.946758 10657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: Generated new TSK 0
I20260925 16:39:09.947672 10657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:39:09.984706 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:10.096868 10669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:10.097157 10669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:10.097201 10669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:10.101321 10669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:10.101387 10669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:10.101492 10669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:39:10.106794 10669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:10.107906 10669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:10.109185 10669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:10.119705 10674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:10.119711 10677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:10.119771 10669 server_base.cc:1034] running on GCE node
W20260925 16:39:10.119705 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:10.120380 10669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:10.121052 10669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:10.122251 10669 hybrid_clock.cc:648] HybridClock initialized: now 1790354350122241 us; error 57 us; skew 500 ppm
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:10.125481 10669 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:39:10.126415 10669 webserver.cc:460] Webserver started at http://127.0.97.193:42611/ using document root <none> and password file <none>
I20260925 16:39:10.126968 10669 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:10.127034 10669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:10.127207 10669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:10.129387 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "768cd813b22545dcbd0498921ea8e4fe"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "ceaf3e91fb97e4d2b4472c97f34117f6"
server_key_iv: "f2f167afe6d35777f2655d8721b6cd8d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.129901 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "768cd813b22545dcbd0498921ea8e4fe"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "ceaf3e91fb97e4d2b4472c97f34117f6"
server_key_iv: "f2f167afe6d35777f2655d8721b6cd8d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.133723 10669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 16:39:10.136310 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.137676 10669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:39:10.137773 10669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "768cd813b22545dcbd0498921ea8e4fe"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "ceaf3e91fb97e4d2b4472c97f34117f6"
server_key_iv: "f2f167afe6d35777f2655d8721b6cd8d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.137859 10669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:10.156102 10669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:10.158962 10669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:10.159123 10669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:10.159631 10669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:10.160389 10669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:10.160436 10669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.160476 10669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:10.160497 10669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.169068 10669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:33817
I20260925 16:39:10.169092 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:33817 every 8 connection(s)
I20260925 16:39:10.169992 10669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:39:10.171425 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10669
I20260925 16:39:10.171535 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:39:10.171859 391 external_mini_cluster.cc:1428] Setting key e48514bbd1bdcef89e6d06bdd96b3ddc
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:39:10.185535 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.171824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:42689 (local address 127.0.97.254:41531)
0925 16:39:10.171964 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.171968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.173523 (+ 1555us) server_negotiation.cc:407] Connection header received
0925 16:39:10.174886 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.174889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.174931 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.175007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:10.176263 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.176828 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.178715 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.178939 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.179021 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.181922 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:10.181926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:10.181929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:10.181963 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:10.183415 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.183917 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.183920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.183922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.184002 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.184345 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.184349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.184350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.184541 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:10.184656 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.185242 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.185407 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:39:10.186643 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.172288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41531 (local address 127.0.97.193:42689)
0925 16:39:10.173362 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.173422 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.174657 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.175229 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.175240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.175646 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:10.176114 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.176136 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.176970 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.176976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.178578 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.178593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.179105 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.179112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.179354 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.180005 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:10.180028 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:10.181756 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:10.183532 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.183538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.183553 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.183809 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.184099 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.184103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.184105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.184243 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.184652 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:10.184658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:10.184941 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.185162 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.185560 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":651,"threads_started":1}
I20260925 16:39:10.187912 10798 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41531
I20260925 16:39:10.188174 10798 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:10.188661 10798 heartbeater.cc:511] Master 127.0.97.254:41531 requested a full tablet report, sending...
I20260925 16:39:10.190253 10607 ts_manager.cc:194] Registered new tserver with Master: 768cd813b22545dcbd0498921ea8e4fe (127.0.97.193:33817)
I20260925 16:39:10.191632 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:42689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:39:10.201099 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:10.313732 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:10.313985 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:10.314062 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:10.318126 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:10.318202 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:10.318296 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:39:10.323529 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:10.324569 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:10.325610 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:10.335091 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:10.335088 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:10.335067 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:10.335287 10803 server_base.cc:1034] running on GCE node
I20260925 16:39:10.335840 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:10.336452 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:10.337657 10803 hybrid_clock.cc:648] HybridClock initialized: now 1790354350337631 us; error 52 us; skew 500 ppm
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:10.340294 10803 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:39:10.341176 10803 webserver.cc:460] Webserver started at http://127.0.97.194:44427/ using document root <none> and password file <none>
I20260925 16:39:10.341789 10803 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:10.341851 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:10.342059 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:10.343959 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "03203dd61bca46d7b1359c313afe81b4"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "a8a95c4d929c9d55aac9f333ab713304"
server_key_iv: "faeec092c08016368bd56b84f4112a68"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.344451 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "03203dd61bca46d7b1359c313afe81b4"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "a8a95c4d929c9d55aac9f333ab713304"
server_key_iv: "faeec092c08016368bd56b84f4112a68"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.347957 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:39:10.350229 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.351377 10803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:10.351469 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "03203dd61bca46d7b1359c313afe81b4"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "a8a95c4d929c9d55aac9f333ab713304"
server_key_iv: "faeec092c08016368bd56b84f4112a68"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.351547 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:10.375371 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:10.378204 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:10.378356 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:10.378871 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:10.379612 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:10.379657 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.379695 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:10.379715 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.388474 10803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:36599
I20260925 16:39:10.388499 10931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:36599 every 8 connection(s)
I20260925 16:39:10.389410 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:39:10.397503 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10803
I20260925 16:39:10.397629 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:39:10.397905 391 external_mini_cluster.cc:1428] Setting key 82837667b8b6b77f80e3d919815b192e
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:39:10.403605 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.391120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:53595 (local address 127.0.97.254:41531)
0925 16:39:10.391253 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.391256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.392461 (+ 1205us) server_negotiation.cc:407] Connection header received
0925 16:39:10.393514 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.393517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.393574 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.393654 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:10.394784 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.395319 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.396434 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.396671 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.396781 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.399713 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:10.399716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:10.399717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:10.399740 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:10.401456 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.401928 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.401934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.401939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.402030 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.402452 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.402454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.402455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.402629 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:10.402704 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.403323 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.403479 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:39:10.404834 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.391600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41531 (local address 127.0.97.194:53595)
0925 16:39:10.392299 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.392339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.393258 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.393820 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.393826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.394177 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:10.394633 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.394655 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.395446 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.395450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.396309 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.396318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.396818 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.396823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.396986 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.397885 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:10.397900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:10.399546 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:10.401564 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.401568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.401578 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.401827 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.402161 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.402167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.402172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.402357 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.402735 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:10.402745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:10.403061 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.403293 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.403663 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":145,"threads_started":1}
I20260925 16:39:10.406039 10932 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41531
I20260925 16:39:10.406279 10932 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:10.406754 10932 heartbeater.cc:511] Master 127.0.97.254:41531 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:39:10.407809 10607 ts_manager.cc:194] Registered new tserver with Master: 03203dd61bca46d7b1359c313afe81b4 (127.0.97.194:36599)
I20260925 16:39:10.408384 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:53595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:39:10.425685 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:10.536788 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:10.537058 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:10.537138 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:10.541117 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:10.541180 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:10.541239 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:39:10.546602 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:10.547606 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:10.548548 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:10.558002 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:10.558025 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:10.558195 10937 server_base.cc:1034] running on GCE node
W20260925 16:39:10.558041 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:10.558727 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:10.559334 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:10.560488 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790354350560476 us; error 29 us; skew 500 ppm
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:10.563125 10937 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:39:10.563912 10937 webserver.cc:460] Webserver started at http://127.0.97.195:33199/ using document root <none> and password file <none>
I20260925 16:39:10.564411 10937 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:10.564466 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:10.564625 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:39:10.566629 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "ccc597306f664836a543aaf117d2e534"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "830f38c10e59544ab1e84003dfe53c3d"
server_key_iv: "9e885070a484b15e601f51999aff952d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.567135 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "ccc597306f664836a543aaf117d2e534"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "830f38c10e59544ab1e84003dfe53c3d"
server_key_iv: "9e885070a484b15e601f51999aff952d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.570755 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:39:10.573097 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.574358 10937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:39:10.574448 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "ccc597306f664836a543aaf117d2e534"
format_stamp: "Formatted at 2026-09-25 16:39:10 on dist-test-slave-kxl8"
server_key: "830f38c10e59544ab1e84003dfe53c3d"
server_key_iv: "9e885070a484b15e601f51999aff952d"
server_key_version: "encryptionkey@0"
I20260925 16:39:10.574529 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:10.586867 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:10.590092 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:10.590293 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:10.590842 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:10.591745 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:10.591799 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.591837 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:10.591858 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:10.600641 10937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:46607
I20260925 16:39:10.600703 11065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:46607 every 8 connection(s)
I20260925 16:39:10.601608 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:39:10.611415 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 10937
I20260925 16:39:10.611555 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:39:10.611948 391 external_mini_cluster.cc:1428] Setting key a92512eb24737e609bc26a29f5cf1617
I20260925 16:39:10.614897 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.603427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:53895 (local address 127.0.97.254:41531)
0925 16:39:10.603555 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.603559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.604594 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 16:39:10.605565 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.605569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.605626 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.605735 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:10.606882 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.607412 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.608535 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.608801 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.608910 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.611296 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:10.611302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:10.611307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:10.611337 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:10.612899 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.613428 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.613432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.613434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.613513 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.613880 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.613882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.613883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.614114 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:10.614230 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.614690 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.614784 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:39:10.615717 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.603815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:41531 (local address 127.0.97.195:53895)
0925 16:39:10.604437 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.604477 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.605350 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.605907 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.605914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.606269 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:10.606740 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.606753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.607538 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.607541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.608402 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.608411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.608943 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.608947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.609129 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.609705 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:10.609721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:10.611124 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:10.612997 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.613001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.613012 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.613323 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.613608 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:10.613612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:10.613613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:10.613784 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:10.614245 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:10.614250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:10.614469 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.614627 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.614866 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":108,"threads_started":1}
I20260925 16:39:10.616812 11066 heartbeater.cc:347] Connected to a master server at 127.0.97.254:41531
I20260925 16:39:10.617017 11066 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:10.617498 11066 heartbeater.cc:511] Master 127.0.97.254:41531 requested a full tablet report, sending...
I20260925 16:39:10.618420 10607 ts_manager.cc:194] Registered new tserver with Master: ccc597306f664836a543aaf117d2e534 (127.0.97.195:46607)
I20260925 16:39:10.618968 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:53895
I20260925 16:39:10.625697 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:39:10.634258 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.627540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.97.254:41531)
0925 16:39:10.627687 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.627694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.627797 (+ 103us) server_negotiation.cc:407] Connection header received
0925 16:39:10.627896 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.627898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.627931 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.627994 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:10.628662 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.629158 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.629989 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.630191 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.630268 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.631597 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:10.631599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:10.631601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:10.631617 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:10.632802 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.633157 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.633159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.633160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.633229 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:10.633500 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:10.633503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:10.633504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:10.633707 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:10.633806 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.633991 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.634076 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:39:10.636914 10607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:39:10.639909 10607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:39:10.654356 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.648225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36599 (local address 127.0.0.1:44778)
0925 16:39:10.649098 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.649113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.649204 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.650083 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.650089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.650122 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:10.650350 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.650358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651430 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.651435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.652684 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.652693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.653808 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.653812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.653935 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.653964 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.654024 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.654198 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":110,"spinlock_wait_cycles":2432,"thread_start_us":94,"threads_started":1}
I20260925 16:39:10.654398 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.649158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.97.193:33817)
0925 16:39:10.649874 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.649889 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.649912 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:39:10.649982 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.649991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.650172 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.650304 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:10.650697 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651808 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.652980 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.653635 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.653767 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.653920 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.654071 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.654223 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":113,"threads_started":1}
I20260925 16:39:10.654626 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.97.194:36599)
0925 16:39:10.649598 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.649610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.649633 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:39:10.649710 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.649717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.649925 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.650094 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:10.650484 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651302 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.652801 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.653675 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.654221 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.654350 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.654396 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.654477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
I20260925 16:39:10.655172 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.648876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:33817 (local address 127.0.0.1:42620)
0925 16:39:10.649684 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.649697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.649780 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.650461 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.650465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.650491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:10.650576 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.650581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651939 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.651943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.652852 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.652861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.653767 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.653771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.653871 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.653907 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.654042 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.655008 (+ 966us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":34,"thread_start_us":100,"threads_started":1}
I20260925 16:39:10.655653 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.648122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:46607 (local address 127.0.0.1:40856)
0925 16:39:10.648463 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.648510 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.648630 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.650089 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.650092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.650127 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:10.650350 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.650358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651764 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.651767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.653852 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.653860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.655202 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.655208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.655340 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.655361 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.655454 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.655542 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":947,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260925 16:39:10.656243 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.648977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.97.195:46607)
0925 16:39:10.649425 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.649438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.649464 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:39:10.649546 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.649553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.649939 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.650281 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:10.650456 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.651608 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.653972 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.654867 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.654997 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.655125 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.655921 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.656015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":104,"threads_started":1}
I20260925 16:39:10.657941 10866 tablet_service.cc:1467] Processing CreateTablet for tablet 73426fd55d604439b7e8f98866304f09 (DEFAULT_TABLE table=test-table [id=3d234ffaffa64e94a0ddafb254021b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:10.657967 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 73426fd55d604439b7e8f98866304f09 (DEFAULT_TABLE table=test-table [id=3d234ffaffa64e94a0ddafb254021b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:10.658931 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73426fd55d604439b7e8f98866304f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:10.658975 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73426fd55d604439b7e8f98866304f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:10.659768 11000 tablet_service.cc:1467] Processing CreateTablet for tablet 73426fd55d604439b7e8f98866304f09 (DEFAULT_TABLE table=test-table [id=3d234ffaffa64e94a0ddafb254021b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:10.660698 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73426fd55d604439b7e8f98866304f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:10.664829 11083 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Bootstrap starting.
I20260925 16:39:10.666301 11084 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: Bootstrap starting.
I20260925 16:39:10.667287 11085 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Bootstrap starting.
I20260925 16:39:10.668219 11084 tablet_bootstrap.cc:654] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:10.668380 11083 tablet_bootstrap.cc:654] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:10.669248 11083 log.cc:824] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:10.669261 11084 log.cc:824] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:10.669454 11085 tablet_bootstrap.cc:654] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:10.670593 11085 log.cc:824] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:10.671361 11084 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: No bootstrap required, opened a new log
I20260925 16:39:10.671361 11083 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: No bootstrap required, opened a new log
I20260925 16:39:10.671553 11084 ts_tablet_manager.cc:1397] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 16:39:10.671553 11083 ts_tablet_manager.cc:1397] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:39:10.672394 11085 tablet_bootstrap.cc:492] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: No bootstrap required, opened a new log
I20260925 16:39:10.672590 11085 ts_tablet_manager.cc:1397] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:39:10.675372 11085 raft_consensus.cc:348] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.675410 11084 raft_consensus.cc:348] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.675410 11083 raft_consensus.cc:348] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.675637 11085 raft_consensus.cc:374] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:10.675652 11083 raft_consensus.cc:374] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:10.675659 11084 raft_consensus.cc:374] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:10.675709 11085 raft_consensus.cc:729] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc597306f664836a543aaf117d2e534, State: Initialized, Role: FOLLOWER
I20260925 16:39:10.675715 11084 raft_consensus.cc:729] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768cd813b22545dcbd0498921ea8e4fe, State: Initialized, Role: FOLLOWER
I20260925 16:39:10.675743 11083 raft_consensus.cc:729] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03203dd61bca46d7b1359c313afe81b4, State: Initialized, Role: FOLLOWER
I20260925 16:39:10.676172 11085 consensus_queue.cc:260] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.676198 11083 consensus_queue.cc:260] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.676194 11084 consensus_queue.cc:260] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.676829 11066 heartbeater.cc:503] Master 127.0.97.254:41531 was elected leader, sending a full tablet report...
I20260925 16:39:10.677136 11085 ts_tablet_manager.cc:1428] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 16:39:10.677140 11083 ts_tablet_manager.cc:1428] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:39:10.677140 11084 ts_tablet_manager.cc:1428] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:39:10.677479 10932 heartbeater.cc:503] Master 127.0.97.254:41531 was elected leader, sending a full tablet report...
I20260925 16:39:10.677479 10798 heartbeater.cc:503] Master 127.0.97.254:41531 was elected leader, sending a full tablet report...
I20260925 16:39:10.805721 11090 raft_consensus.cc:482] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:39:10.805955 11090 raft_consensus.cc:504] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.807109 11090 leader_election.cc:288] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccc597306f664836a543aaf117d2e534 (127.0.97.195:46607), 03203dd61bca46d7b1359c313afe81b4 (127.0.97.194:36599)
I20260925 16:39:10.811313 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.807489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:56219 (local address 127.0.97.194:36599)
0925 16:39:10.807627 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.807632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.807864 (+ 232us) server_negotiation.cc:407] Connection header received
0925 16:39:10.807968 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.807974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.808033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.808112 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:10.808403 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.809018 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810299 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.810692 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810819 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.810873 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.811121 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.811190 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:39:10.811337 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.807365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:46607 (local address 127.0.97.193:45101)
0925 16:39:10.807902 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.807916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.808033 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.808317 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.808319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.808333 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:10.808467 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.808474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.809227 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.809230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.810167 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.810176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.810814 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.811008 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.811026 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.811129 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.811194 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":142,"thread_start_us":117,"threads_started":1}
I20260925 16:39:10.811348 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.807579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:45101 (local address 127.0.97.195:46607)
0925 16:39:10.807702 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.807706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.808017 (+ 311us) server_negotiation.cc:407] Connection header received
0925 16:39:10.808164 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.808167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.808216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.808289 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:10.808571 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.809117 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810299 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.810691 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810818 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.810873 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.811174 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.811248 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:39:10.811342 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.807365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36599 (local address 127.0.97.193:56219)
0925 16:39:10.807769 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.807781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:10.807878 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:10.808140 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:10.808143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:10.808160 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:10.808293 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.808300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.809119 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.809123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.810170 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:10.810179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.810815 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.810818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:10.810990 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.811013 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:39:10.811073 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:39:10.811166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":93,"thread_start_us":105,"threads_started":1}
I20260925 16:39:10.812075 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73426fd55d604439b7e8f98866304f09" candidate_uuid: "768cd813b22545dcbd0498921ea8e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc597306f664836a543aaf117d2e534" is_pre_election: true
I20260925 16:39:10.812193 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73426fd55d604439b7e8f98866304f09" candidate_uuid: "768cd813b22545dcbd0498921ea8e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03203dd61bca46d7b1359c313afe81b4" is_pre_election: true
I20260925 16:39:10.812441 11020 raft_consensus.cc:2454] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768cd813b22545dcbd0498921ea8e4fe in term 0.
I20260925 16:39:10.812516 10886 raft_consensus.cc:2454] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768cd813b22545dcbd0498921ea8e4fe in term 0.
I20260925 16:39:10.812928 10688 leader_election.cc:302] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768cd813b22545dcbd0498921ea8e4fe, ccc597306f664836a543aaf117d2e534; no voters:
I20260925 16:39:10.813211 11090 raft_consensus.cc:2781] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:39:10.813311 11090 raft_consensus.cc:482] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:39:10.813383 11090 raft_consensus.cc:3037] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:10.814416 11090 raft_consensus.cc:504] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.814792 11090 leader_election.cc:288] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [CANDIDATE]: Term 1 election: Requested vote from peers ccc597306f664836a543aaf117d2e534 (127.0.97.195:46607), 03203dd61bca46d7b1359c313afe81b4 (127.0.97.194:36599)
I20260925 16:39:10.815114 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73426fd55d604439b7e8f98866304f09" candidate_uuid: "768cd813b22545dcbd0498921ea8e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc597306f664836a543aaf117d2e534"
I20260925 16:39:10.815207 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73426fd55d604439b7e8f98866304f09" candidate_uuid: "768cd813b22545dcbd0498921ea8e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03203dd61bca46d7b1359c313afe81b4"
I20260925 16:39:10.815270 11020 raft_consensus.cc:3037] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:10.815346 10886 raft_consensus.cc:3037] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:10.816275 11020 raft_consensus.cc:2454] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768cd813b22545dcbd0498921ea8e4fe in term 1.
I20260925 16:39:10.816370 10886 raft_consensus.cc:2454] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768cd813b22545dcbd0498921ea8e4fe in term 1.
I20260925 16:39:10.816542 10688 leader_election.cc:302] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768cd813b22545dcbd0498921ea8e4fe, ccc597306f664836a543aaf117d2e534; no voters:
I20260925 16:39:10.816752 11090 raft_consensus.cc:2781] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:10.816967 11090 raft_consensus.cc:686] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [term 1 LEADER]: Becoming Leader. State: Replica: 768cd813b22545dcbd0498921ea8e4fe, State: Running, Role: LEADER
I20260925 16:39:10.817337 11090 consensus_queue.cc:237] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } }
I20260925 16:39:10.820556 10607 catalog_manager.cc:5495] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 768cd813b22545dcbd0498921ea8e4fe (127.0.97.193). New cstate: current_term: 1 leader_uuid: "768cd813b22545dcbd0498921ea8e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768cd813b22545dcbd0498921ea8e4fe" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33817 } health_report { overall_health: HEALTHY } } }
I20260925 16:39:10.843086 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.839490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.0.97.193:33817)
0925 16:39:10.839620 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:10.839624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:10.839637 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:39:10.839765 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:10.839768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:10.839822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:10.839911 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:39:10.840113 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.840596 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.841692 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:10.841964 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:10.842046 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:10.842309 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:39:10.842719 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:39:10.842809 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:10.842868 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 16:39:10.842913 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:39:10.849097 10886 raft_consensus.cc:1261] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 1 FOLLOWER]: Refusing update from remote peer 768cd813b22545dcbd0498921ea8e4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:39:10.849113 11020 raft_consensus.cc:1261] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 1 FOLLOWER]: Refusing update from remote peer 768cd813b22545dcbd0498921ea8e4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:39:10.849879 11090 consensus_queue.cc:1035] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc597306f664836a543aaf117d2e534" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:39:10.850081 11094 consensus_queue.cc:1035] T 73426fd55d604439b7e8f98866304f09 P 768cd813b22545dcbd0498921ea8e4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "03203dd61bca46d7b1359c313afe81b4" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:39:10.852496 11067 tablet.cc:2369] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:39:10.857993 11098 mvcc.cc:205] Tried to move back new op lower bound from 7333291421070495744 to 7333291420950601728. Current Snapshot: MvccSnapshot[applied={T|T < 7333291421070495744}]
I20260925 16:39:10.859203 11099 mvcc.cc:205] Tried to move back new op lower bound from 7333291421070495744 to 7333291420950601728. Current Snapshot: MvccSnapshot[applied={T|T < 7333291421070495744}]
I20260925 16:39:10.870303 10607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:39:10.870433 10607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:39:10.872814 10607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1286c922e4047578c86186e5da69764: Sending DeleteTablet for 3 replicas of tablet 73426fd55d604439b7e8f98866304f09
I20260925 16:39:10.873700 10866 tablet_service.cc:1514] Processing DeleteTablet for tablet 73426fd55d604439b7e8f98866304f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:10 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44778
I20260925 16:39:10.873728 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet 73426fd55d604439b7e8f98866304f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:10 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:40856
I20260925 16:39:10.874223 11108 tablet_replica.cc:330] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: stopping tablet replica
I20260925 16:39:10.874356 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet 73426fd55d604439b7e8f98866304f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:10 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42620
I20260925 16:39:10.874555 11108 raft_consensus.cc:2229] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:39:10.874838 11108 raft_consensus.cc:2258] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:39:10.875703 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10669
I20260925 16:39:10.877914 11107 tablet_replica.cc:330] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: stopping tablet replica
I20260925 16:39:10.878316 11107 raft_consensus.cc:2229] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:39:10.878679 11107 raft_consensus.cc:2258] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:39:10.878750 11108 ts_tablet_manager.cc:1905] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:39:10.880182 11107 ts_tablet_manager.cc:1905] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:39:10.881748 11108 ts_tablet_manager.cc:1918] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:39:10.881904 11108 log.cc:1196] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/73426fd55d604439b7e8f98866304f09
I20260925 16:39:10.882277 11108 ts_tablet_manager.cc:1939] T 73426fd55d604439b7e8f98866304f09 P ccc597306f664836a543aaf117d2e534: Deleting consensus metadata
I20260925 16:39:10.883467 10594 catalog_manager.cc:4841] TS ccc597306f664836a543aaf117d2e534 (127.0.97.195:46607): tablet 73426fd55d604439b7e8f98866304f09 (table test-table [id=3d234ffaffa64e94a0ddafb254021b50]) successfully deleted
I20260925 16:39:10.883832 11107 ts_tablet_manager.cc:1918] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 16:39:10.883924 10593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:33817 (error 108)
I20260925 16:39:10.883981 11107 log.cc:1196] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/73426fd55d604439b7e8f98866304f09
I20260925 16:39:10.884330 11107 ts_tablet_manager.cc:1939] T 73426fd55d604439b7e8f98866304f09 P 03203dd61bca46d7b1359c313afe81b4: Deleting consensus metadata
I20260925 16:39:10.884814 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10803
I20260925 16:39:10.884935 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.884646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:33817 (local address 127.0.0.1:42632)
0925 16:39:10.884775 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.884840 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:39:10.885792 10593 catalog_manager.cc:4584] TS 768cd813b22545dcbd0498921ea8e4fe (127.0.97.193:33817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73426fd55d604439b7e8f98866304f09: Network error: Client connection negotiation failed: client connection to 127.0.97.193:33817: connect: Connection refused (error 111)
I20260925 16:39:10.890319 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10937
I20260925 16:39:10.891409 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:10.890537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:36599 (local address 127.0.0.1:44788)
0925 16:39:10.891250 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:10.891313 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612}
I20260925 16:39:10.895956 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 10575
2026-09-25T16:39:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2385 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[11113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[11113](info): set up 2 sockets
Sep 25 16:39:10 dist-test-slave-kxl8 krb5kdc[11113](info): commencing operation
krb5kdc: starting...
W20260925 16:39:11.954248 391 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:39:12 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:39:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangeradmin_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangeradmin_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangerlookup_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangerlookup_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
I20260925 16:39:12.114468 391 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVEDvT3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/postgres -l logfile start
2026-09-25 16:39:14.686 UTC [11143] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:39:14.686 UTC [11143] LOG: listening on IPv4 address "127.0.97.212", port 42753
2026-09-25 16:39:14.690 UTC [11143] LOG: listening on Unix socket "/tmp/.s.PGSQL.42753"
2026-09-25 16:39:14.712 UTC [11146] LOG: database system was shut down at 2026-09-25 16:39:13 UTC
2026-09-25 16:39:14.716 UTC [11143] LOG: database system is ready to accept connections
I20260925 16:39:15.670138 391 mini_postgres.cc:96] Postgres bound to 42753
2026-09-25 16:39:15.676 UTC [11154] FATAL: database "slave" does not exist
127.0.97.212:42753 - accepting connections
I20260925 16:39:15.677330 391 mini_ranger.cc:161] Starting Ranger...
I20260925 16:39:15.691967 391 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:39:16.176031 391 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:39:16.176129 391 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin
2026-09-25 16:39:16,459 [I] DB FLAVOR :POSTGRES
2026-09-25 16:39:16,460 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:39:16,460 [I] Checking connection..
2026-09-25 16:39:16,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:39:16,712 [I] Checking connection passed.
2026-09-25 16:39:16,712 [I] --------- Verifying version history table ---------
2026-09-25 16:39:16,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:39:16,974 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:39:16,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:39:17,201 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:39:17,201 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:39:17,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:39:17.453 UTC [11218] WARNING: there is no transaction in progress
2026-09-25 16:39:17,460 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:39:17,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:39:17,685 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:39:17,685 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:39:17,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:39:17,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:39:18,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kxl8.c.gcp-upstream.internal','N') ;"
2026-09-25 16:39:18,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:39:18,673 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:39:18,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:39:18,932 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:39:18,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:39:19,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:39:19,439 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:39:19,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:39:20.386 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.402 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.418 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.433 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.449 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.466 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.523 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.529 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.543 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.553 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.563 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.576 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.587 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.593 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.600 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:20.607 UTC [11326] WARNING: there is no transaction in progress
2026-09-25 16:39:21,232 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:39:21,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:39:21,477 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:39:21,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:39:21,737 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:39:21,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:39:21,984 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:39:21,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:39:22,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:39:22,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kxl8.c.gcp-upstream.internal';"
2026-09-25 16:39:22,690 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:39:22,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:39:22,962 [I] DB_PATCHES have already been applied
I20260925 16:39:22.968339 391 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVEDvT3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:39:22.968444 391 mini_ranger.cc:193] Using host: 127.0.97.212
I20260925 16:39:22.971964 391 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.212:47827
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:39:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:39:24.065717 391 mini_ranger.cc:160] Time spent starting Ranger: real 8.388s user 0.000s sys 0.010s
Sep 25, 2026 4:39:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:39:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 4:39:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:39:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:39:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:39:45.945179 391 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangerkms_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/rangerkms_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:39:46 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354386, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:39:46.030731 391 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:39:46.040588 391 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:39:46.938050 391 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:39:46.938143 391 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms
I20260925 16:39:46.938243 391 mini_ranger_kms.cc:227] Using postgres at 127.0.97.212:42753
2026-09-25 16:39:47,122 [I] DB FLAVOR :POSTGRES
2026-09-25 16:39:47,122 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:39:47,122 [I] Checking connection
2026-09-25 16:39:47,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:39:47,353 [I] connection success
2026-09-25 16:39:47,354 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:39:47,354 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:39:47,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:39:47,583 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:39:47,583 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:39:47,583 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:39:47,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:42753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:39:47,849 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:39:48.326025 391 mini_ranger_kms.cc:310] Created kms service
I20260925 16:39:48.502125 391 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:39:48.549839 391 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:39:48 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354386, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:48.952805 391 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:39:48.952972 391 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:39:48.953003 391 mini_ranger_kms.cc:240] Using host: 127.0.97.212
I20260925 16:39:48.956976 391 mini_ranger_kms.cc:276] Ranger KMS PID: 11506
I20260925 16:39:48.957115 391 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.212:34643
Sep 25, 2026 4:39:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:39:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:39:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:39:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:39:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:39:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 16:39:51 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354391, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:51.241214 391 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.210s user 0.001s sys 0.168s
I20260925 16:39:51.241430 391 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:39:51.241549 391 mini_ranger_kms.cc:401] 127.0.97.212:34643/kms/v1/keys
Sep 25 16:39:51 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354391, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
Sep 25 16:39:51 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354391, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
Sep 25 16:39:54 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:39:54.306727 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45949
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:45949
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:34643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:54.436599 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:54.436847 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:54.436890 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:54.440872 11557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:39:54.440927 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:54.440946 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:54.440963 11557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:39:54.441018 11557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:39:54.446416 11557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.212:34643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:45949
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:54.447407 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:54.448400 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:54.456820 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:54.456830 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:54.456954 11557 server_base.cc:1034] running on GCE node
W20260925 16:39:54.456820 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:54.457828 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:54.458992 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:54.460192 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790354394460171 us; error 32 us; skew 500 ppm
Sep 25 16:39:54 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:54.463277 11557 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:39:54.464152 11557 webserver.cc:460] Webserver started at http://127.0.97.254:40987/ using document root <none> and password file <none>
I20260925 16:39:54.464720 11557 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:54.464779 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:54.464952 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:39:54 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:54.525658 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "1ebe51d2e17a4f6b9f04546b97436a1c"
format_stamp: "Formatted at 2026-09-25 16:39:54 on dist-test-slave-kxl8"
server_key: "a1dc627944f45572c44992ceb35aa9d8"
server_key_iv: "e78e12d8e16b46eb1c4d0843035e83ef"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:54.526473 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "1ebe51d2e17a4f6b9f04546b97436a1c"
format_stamp: "Formatted at 2026-09-25 16:39:54 on dist-test-slave-kxl8"
server_key: "a1dc627944f45572c44992ceb35aa9d8"
server_key_iv: "e78e12d8e16b46eb1c4d0843035e83ef"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:54.530527 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 16:39:54.545421 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.001s
I20260925 16:39:54.546871 11557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:39:54.547024 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "1ebe51d2e17a4f6b9f04546b97436a1c"
format_stamp: "Formatted at 2026-09-25 16:39:54 on dist-test-slave-kxl8"
server_key: "a1dc627944f45572c44992ceb35aa9d8"
server_key_iv: "e78e12d8e16b46eb1c4d0843035e83ef"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:54.547179 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:54.567155 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:54.570310 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:54.570544 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:54.578883 11557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:45949
I20260925 16:39:54.578948 11624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:45949 every 8 connection(s)
I20260925 16:39:54.579862 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:39:54.582640 11625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:54.585414 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 11557
I20260925 16:39:54.585547 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:39:54.585826 391 external_mini_cluster.cc:1428] Setting key 8bf648536ede7f58ee63b8e4997083f2
I20260925 16:39:54.588708 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: Bootstrap starting.
I20260925 16:39:54.591293 11625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:54.592114 11625 log.cc:824] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:54.594343 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: No bootstrap required, opened a new log
Sep 25 16:39:54 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:39:54.598742 11625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } }
I20260925 16:39:54.599044 11625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:54.599138 11625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebe51d2e17a4f6b9f04546b97436a1c, State: Initialized, Role: FOLLOWER
I20260925 16:39:54.599677 11625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } }
I20260925 16:39:54.599864 11625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:39:54.599969 11625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:39:54.600085 11625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:54.601263 11625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } }
I20260925 16:39:54.601686 11625 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebe51d2e17a4f6b9f04546b97436a1c; no voters:
I20260925 16:39:54.602068 11625 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:39:54.602283 11630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:54.602545 11630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [term 1 LEADER]: Becoming Leader. State: Replica: 1ebe51d2e17a4f6b9f04546b97436a1c, State: Running, Role: LEADER
I20260925 16:39:54.602857 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:54.587332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.97.254:45949)
0925 16:39:54.587809 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:54.587829 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:54.587870 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:39:54.588669 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:54.588690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:54.589096 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:54.589470 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:54.590322 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:54.591804 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:54.593065 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:54.593771 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:54.593922 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:54.597130 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:54.597137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:54.597157 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:54.597188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:54.599321 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:54.599837 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:54.599842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:54.599848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:54.600012 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:54.600349 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:54.600354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:54.600356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:54.600851 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:54.601032 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:54.601448 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 16:39:54.601906 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
I20260925 16:39:54.602919 11630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } }
I20260925 16:39:54.603377 11625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:39:54.604264 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebe51d2e17a4f6b9f04546b97436a1c. Latest consensus state: current_term: 1 leader_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } } }
I20260925 16:39:54.604375 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:54.605237 11632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebe51d2e17a4f6b9f04546b97436a1c" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45949 } } }
I20260925 16:39:54.605690 11625 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:39:54.606407 11632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c [sys.catalog]: This master's current role is: LEADER
I20260925 16:39:54.606189 11639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:39:54.614078 11639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:39:54.621836 11639 catalog_manager.cc:1347] Generated new cluster ID: dfb3a6fa242d4a1fb53c3dbf4f41a92c
I20260925 16:39:54.622078 11639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:39:54.658752 11639 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:39:54.660130 11639 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:39:54.667464 11639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: Generated new TSK 0
I20260925 16:39:54.668370 11639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:39:55.364540 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:34643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:39:55.516196 11670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:55.516470 11670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:55.516568 11670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:55.520808 11670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:55.520897 11670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:55.520987 11670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:39:55.526352 11670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.212:34643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:55.527417 11670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:55.528527 11670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:55.539837 11675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:55.540135 11678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:55.540271 11670 server_base.cc:1034] running on GCE node
W20260925 16:39:55.540623 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:55.541118 11670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:55.541991 11670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:55.545300 11670 hybrid_clock.cc:648] HybridClock initialized: now 1790354395545244 us; error 53 us; skew 500 ppm
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:55.548682 11670 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:39:55.549681 11670 webserver.cc:460] Webserver started at http://127.0.97.193:43881/ using document root <none> and password file <none>
I20260925 16:39:55.550249 11670 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:55.550335 11670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:55.550612 11670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:55.573688 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "6ee7950c919048da8f77a00266eccf8a"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "a8fdde743875b98331a3c64fc0b34cec"
server_key_iv: "6fcef665859aa18b75d4349f5847479f"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.574488 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "6ee7950c919048da8f77a00266eccf8a"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "a8fdde743875b98331a3c64fc0b34cec"
server_key_iv: "6fcef665859aa18b75d4349f5847479f"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.580163 11670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:39:55.603595 11687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:55.605320 11670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:39:55.605456 11670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "6ee7950c919048da8f77a00266eccf8a"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "a8fdde743875b98331a3c64fc0b34cec"
server_key_iv: "6fcef665859aa18b75d4349f5847479f"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.605600 11670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:55.623638 11670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:55.636195 11670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:55.636382 11670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:55.637071 11670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:55.638176 11670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:55.638226 11670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:55.638275 11670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:55.638296 11670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:55.649421 11670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:33161
I20260925 16:39:55.650722 11670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:39:55.652168 11801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:33161 every 8 connection(s)
I20260925 16:39:55.655350 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 11670
I20260925 16:39:55.655480 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:39:55.655782 391 external_mini_cluster.cc:1428] Setting key 82d7f45e125f93a91b89ec65ea9966c6
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:39:55.701839 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:55.655018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:33179 (local address 127.0.97.254:45949)
0925 16:39:55.661701 (+ 6683us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:55.661717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:55.662201 (+ 484us) server_negotiation.cc:407] Connection header received
0925 16:39:55.663692 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:55.663697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:55.663758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:55.663827 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:55.665908 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:55.666840 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:55.668931 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:55.669306 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:55.669387 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:55.690109 (+ 20722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:55.690114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:55.690117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:55.690166 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:55.692394 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:55.694832 (+ 2438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:55.694836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:55.694838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:55.694942 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:55.695459 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:55.695463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:55.695465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:55.695717 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:55.695808 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:55.701464 (+ 5656us) server_negotiation.cc:299] Negotiation successful
0925 16:39:55.701622 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6570,"thread_start_us":108,"threads_started":1}
I20260925 16:39:55.704077 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:55.655131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45949 (local address 127.0.97.193:33179)
0925 16:39:55.662021 (+ 6890us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:55.662095 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:55.663392 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:55.664329 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:55.664342 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:55.664867 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:55.665711 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:55.665735 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:55.667010 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:55.667015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:55.668788 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:55.668798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:55.669613 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:55.669616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:55.669810 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:55.677226 (+ 7416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:55.677254 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:55.689802 (+ 12548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:55.692587 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:55.692594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:55.692614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:55.693023 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:55.695078 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:55.695081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:55.695083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:55.695332 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:55.695886 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:55.695894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:55.696300 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 16:39:55.702392 (+ 6092us) client_negotiation.cc:240] Negotiation successful
0925 16:39:55.702830 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6629,"thread_start_us":131,"threads_started":1}
I20260925 16:39:55.706060 11802 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45949
I20260925 16:39:55.706332 11802 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:55.706943 11802 heartbeater.cc:511] Master 127.0.97.254:45949 requested a full tablet report, sending...
I20260925 16:39:55.708807 11589 ts_manager.cc:194] Registered new tserver with Master: 6ee7950c919048da8f77a00266eccf8a (127.0.97.193:33161)
I20260925 16:39:55.710448 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:33179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:39:55.725142 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:34643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 16:39:55.884864 11809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:55.885231 11809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:55.885315 11809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:55.891574 11809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:55.891703 11809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:55.891822 11809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:39:55.899667 11809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.212:34643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:55.901182 11809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:55.902593 11809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:55.915195 11818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:55.915721 11816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:55.915910 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:55.916177 11809 server_base.cc:1034] running on GCE node
I20260925 16:39:55.916649 11809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:55.917505 11809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:55.918696 11809 hybrid_clock.cc:648] HybridClock initialized: now 1790354395918674 us; error 35 us; skew 500 ppm
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:55.922289 11809 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:39:55.923334 11809 webserver.cc:460] Webserver started at http://127.0.97.194:37337/ using document root <none> and password file <none>
I20260925 16:39:55.924014 11809 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:55.924100 11809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:55.924320 11809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:39:55 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:55.953393 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "b4a6eb4617df4f4faec088548081a53b"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "c1bc3f870128292507a645f235807fab"
server_key_iv: "334d552b5c931b4ee797ce50a259b734"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.954269 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "b4a6eb4617df4f4faec088548081a53b"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "c1bc3f870128292507a645f235807fab"
server_key_iv: "334d552b5c931b4ee797ce50a259b734"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.959869 11809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:39:55.979803 11825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:55.981591 11809 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 16:39:55.981719 11809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "b4a6eb4617df4f4faec088548081a53b"
format_stamp: "Formatted at 2026-09-25 16:39:55 on dist-test-slave-kxl8"
server_key: "c1bc3f870128292507a645f235807fab"
server_key_iv: "334d552b5c931b4ee797ce50a259b734"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:55.981842 11809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:56.006285 11809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:56.011633 11809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:56.011826 11809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:56.012516 11809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:56.013492 11809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:56.013551 11809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:56.013603 11809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:56.013676 11809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:56.025411 11809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:41961
I20260925 16:39:56.025584 11938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:41961 every 8 connection(s)
I20260925 16:39:56.026623 11809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:39:56.026983 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 11809
I20260925 16:39:56.027107 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:39:56.027446 391 external_mini_cluster.cc:1428] Setting key eb9615ad2b02030f2d8c6fd81faa5581
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:39:56.057828 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.032646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:35873 (local address 127.0.97.254:45949)
0925 16:39:56.032781 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.032787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.033694 (+ 907us) server_negotiation.cc:407] Connection header received
0925 16:39:56.035112 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.035118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.035181 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.035242 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:56.037029 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.037992 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.039854 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.040228 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.040320 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.044806 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:56.044810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:56.044813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:56.044844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:56.046830 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.047379 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.047384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.047388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.047488 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.047926 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.047931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.047934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.048179 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:56.048275 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.057476 (+ 9201us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.057651 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:39:56.059401 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.033028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45949 (local address 127.0.97.194:35873)
0925 16:39:56.033543 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.033599 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.034852 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.035699 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.035712 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.036096 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:56.036834 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.036857 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.038183 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.038187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.039676 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.039688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.040548 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.040552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.040827 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.041514 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:56.041531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:56.044191 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:56.046979 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:56.046985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:56.047001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:56.047267 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:56.047616 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:56.047621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:56.047624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:56.047812 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:56.048330 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:56.048339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:56.048702 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.048915 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.049300 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":185,"threads_started":1}
I20260925 16:39:56.061018 11939 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45949
I20260925 16:39:56.061261 11939 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:56.061877 11939 heartbeater.cc:511] Master 127.0.97.254:45949 requested a full tablet report, sending...
I20260925 16:39:56.063020 11589 ts_manager.cc:194] Registered new tserver with Master: b4a6eb4617df4f4faec088548081a53b (127.0.97.194:41961)
I20260925 16:39:56.063786 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:35873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:39:56.071964 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:34643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
W20260925 16:39:56.248716 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:39:56.249092 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:39:56.249176 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:39:56.255295 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:39:56.255385 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:39:56.255478 11944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:39:56.263736 11944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.212:34643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:39:56.265244 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:39:56.266717 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:39:56.279716 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:56.280339 11944 server_base.cc:1034] running on GCE node
W20260925 16:39:56.279963 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:39:56.280063 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:39:56.281069 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:39:56.281920 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:39:56.283138 11944 hybrid_clock.cc:648] HybridClock initialized: now 1790354396283072 us; error 31 us; skew 500 ppm
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:39:56.286924 11944 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:39:56.287971 11944 webserver.cc:460] Webserver started at http://127.0.97.195:44749/ using document root <none> and password file <none>
I20260925 16:39:56.288691 11944 fs_manager.cc:359] Metadata directory not provided
I20260925 16:39:56.288820 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:39:56.289167 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:56.311667 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "a8c905443f5840b1a7e014b6e71cfcdb"
format_stamp: "Formatted at 2026-09-25 16:39:56 on dist-test-slave-kxl8"
server_key: "c48448edd9e91b6ed64dd5af06b04221"
server_key_iv: "3c28510e1556623ef8acd9653fbecd71"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:56.312448 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "a8c905443f5840b1a7e014b6e71cfcdb"
format_stamp: "Formatted at 2026-09-25 16:39:56 on dist-test-slave-kxl8"
server_key: "c48448edd9e91b6ed64dd5af06b04221"
server_key_iv: "3c28510e1556623ef8acd9653fbecd71"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:56.316819 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 16:39:56.333209 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:56.334935 11944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:39:56.335073 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "a8c905443f5840b1a7e014b6e71cfcdb"
format_stamp: "Formatted at 2026-09-25 16:39:56 on dist-test-slave-kxl8"
server_key: "c48448edd9e91b6ed64dd5af06b04221"
server_key_iv: "3c28510e1556623ef8acd9653fbecd71"
server_key_version: "kuduclusterkey@0"
I20260925 16:39:56.335216 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:39:56.348549 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:39:56.352069 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:39:56.352244 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:39:56.352856 11944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:39:56.353833 11944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:39:56.353888 11944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:56.353935 11944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:39:56.354007 11944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:39:56.364519 11944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:39537
I20260925 16:39:56.364598 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:39537 every 8 connection(s)
I20260925 16:39:56.365751 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:39:56.372578 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 11944
I20260925 16:39:56.372697 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:39:56.372954 391 external_mini_cluster.cc:1428] Setting key eeae62c7f3c33144fc67ff852c9a680b
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:39:56.382488 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.368053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:53065 (local address 127.0.97.254:45949)
0925 16:39:56.368186 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.368191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.369262 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 16:39:56.370525 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.370529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.370587 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.370689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:56.372022 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.372908 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.374925 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.375290 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.375368 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.378433 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:56.378437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:56.378440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:56.378467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:56.380171 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.380764 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.380768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.380770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.380856 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.381259 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.381263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.381264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.381507 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:56.381613 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.382199 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.382319 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:39:56.383493 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.368561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45949 (local address 127.0.97.195:53065)
0925 16:39:56.369099 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.369161 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.370273 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.370875 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.370883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.371224 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:39:56.371851 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.371869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.373079 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.373083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.374776 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.374789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.375574 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.375578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.375770 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.376496 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:39:56.376515 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:39:56.378257 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:39:56.380287 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:56.380293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:56.380308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:56.380644 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:56.380957 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:39:56.380960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:39:56.380963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:39:56.381139 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:39:56.381617 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:39:56.381624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:39:56.381910 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.382097 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.382408 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260925 16:39:56.384711 12073 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45949
I20260925 16:39:56.384954 12073 heartbeater.cc:464] Registering TS with master...
I20260925 16:39:56.385561 12073 heartbeater.cc:511] Master 127.0.97.254:45949 requested a full tablet report, sending...
I20260925 16:39:56.386767 11589 ts_manager.cc:194] Registered new tserver with Master: a8c905443f5840b1a7e014b6e71cfcdb (127.0.97.195:39537)
I20260925 16:39:56.387511 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:53065
I20260925 16:39:56.398602 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:39:56.409898 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.400908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.0.97.254:45949)
0925 16:39:56.401024 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.401028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.401563 (+ 535us) server_negotiation.cc:407] Connection header received
0925 16:39:56.401697 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.401700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.401761 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.401843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:39:56.402702 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.403552 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.404742 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.404985 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.405099 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.406684 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:39:56.406687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:39:56.406690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:39:56.406716 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:39:56.408175 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.408617 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.408622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.408624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.408705 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:39:56.408980 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:39:56.408984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:39:56.408985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:39:56.409177 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:39:56.409354 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.409589 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.409704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:39:56.413203 11589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:39:56.417897 11589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:39:56.437377 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.427158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39537 (local address 127.0.0.1:43236)
0925 16:39:56.430788 (+ 3630us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.430825 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.430927 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.433080 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.433083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.433098 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:56.433183 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.433188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.434633 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.434636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.435757 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.435766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.436839 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.436843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.436941 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.436975 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.437073 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.437194 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3476,"thread_start_us":95,"threads_started":1}
I20260925 16:39:56.438181 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.428009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.97.195:39537)
0925 16:39:56.432654 (+ 4645us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.432665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.432687 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:39:56.432756 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.432761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.432930 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.433339 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:56.433460 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.434511 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.435912 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.436704 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.437758 (+ 1054us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.437903 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.437953 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.438036 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4538,"thread_start_us":108,"threads_started":1}
I20260925 16:39:56.438838 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.427996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.0.97.194:41961)
0925 16:39:56.431528 (+ 3532us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.431542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.431570 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:39:56.431649 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.431656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.431884 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.432068 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:56.432355 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.433521 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.435887 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.436704 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.436817 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.436967 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.437446 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.437816 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3403,"thread_start_us":106,"threads_started":1}
I20260925 16:39:56.443536 11995 tablet_service.cc:1467] Processing CreateTablet for tablet 28541e5485c646df8329c9bc53980a00 (DEFAULT_TABLE table=test-table [id=15cce40afe994aa08afe5c5f224108ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:56.444576 11995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28541e5485c646df8329c9bc53980a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:56.437376 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.427233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41961 (local address 127.0.0.1:37636)
0925 16:39:56.431166 (+ 3933us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.431181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.431260 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.432053 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.432058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.432081 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:56.432221 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.432228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.433642 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.433645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.435726 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.435736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.437034 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.437038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.437122 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.437139 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.437201 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.437249 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":82,"threads_started":1}
I20260925 16:39:56.453154 12091 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Bootstrap starting.
I20260925 16:39:56.456385 11855 tablet_service.cc:1467] Processing CreateTablet for tablet 28541e5485c646df8329c9bc53980a00 (DEFAULT_TABLE table=test-table [id=15cce40afe994aa08afe5c5f224108ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:56.458254 11855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28541e5485c646df8329c9bc53980a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:56.460206 12091 tablet_bootstrap.cc:654] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:56.467765 12091 log.cc:824] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:56.472973 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.429123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.97.193:33161)
0925 16:39:56.467842 (+ 38719us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.467850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.467879 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:39:56.467964 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.467969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.468178 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.468725 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:56.468869 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.470025 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.471660 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.472412 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.472552 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.472700 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.472762 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.472847 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38591,"thread_start_us":85,"threads_started":1}
I20260925 16:39:56.472961 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.428744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:33161 (local address 127.0.0.1:34844)
0925 16:39:56.432306 (+ 3562us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.432319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.432390 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.468399 (+ 36009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.468403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.468425 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:56.468560 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.468568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.470164 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.470168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.471500 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.471511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.472554 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.472558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.472640 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.472663 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.472727 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.472808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"thread_start_us":79,"threads_started":1}
I20260925 16:39:56.474211 12092 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Bootstrap starting.
I20260925 16:39:56.476389 11734 tablet_service.cc:1467] Processing CreateTablet for tablet 28541e5485c646df8329c9bc53980a00 (DEFAULT_TABLE table=test-table [id=15cce40afe994aa08afe5c5f224108ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:39:56.477532 11734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28541e5485c646df8329c9bc53980a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:39:56.478073 12092 tablet_bootstrap.cc:654] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:56.479058 12092 log.cc:824] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:56.481065 12092 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: No bootstrap required, opened a new log
I20260925 16:39:56.481302 12092 ts_tablet_manager.cc:1397] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:39:56.482743 12091 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: No bootstrap required, opened a new log
I20260925 16:39:56.483028 12091 ts_tablet_manager.cc:1397] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20260925 16:39:56.486135 12092 raft_consensus.cc:348] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.486421 12092 raft_consensus.cc:374] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:56.486536 12092 raft_consensus.cc:729] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a6eb4617df4f4faec088548081a53b, State: Initialized, Role: FOLLOWER
I20260925 16:39:56.487046 12092 consensus_queue.cc:260] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.488287 12092 ts_tablet_manager.cc:1428] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:39:56.489746 11939 heartbeater.cc:503] Master 127.0.97.254:45949 was elected leader, sending a full tablet report...
I20260925 16:39:56.490173 12096 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Bootstrap starting.
I20260925 16:39:56.495163 12096 tablet_bootstrap.cc:654] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Neither blocks nor log segments found. Creating new log.
I20260925 16:39:56.495522 12091 raft_consensus.cc:348] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.495848 12091 raft_consensus.cc:374] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:56.495910 12091 raft_consensus.cc:729] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c905443f5840b1a7e014b6e71cfcdb, State: Initialized, Role: FOLLOWER
I20260925 16:39:56.496047 12096 log.cc:824] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:39:56.496403 12091 consensus_queue.cc:260] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.497519 12091 ts_tablet_manager.cc:1428] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260925 16:39:56.498296 12096 tablet_bootstrap.cc:492] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: No bootstrap required, opened a new log
I20260925 16:39:56.498487 12096 ts_tablet_manager.cc:1397] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
Sep 25 16:39:56 dist-test-slave-kxl8 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:39:56.498710 12073 heartbeater.cc:503] Master 127.0.97.254:45949 was elected leader, sending a full tablet report...
I20260925 16:39:56.503499 12096 raft_consensus.cc:348] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.503809 12096 raft_consensus.cc:374] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:39:56.503867 12096 raft_consensus.cc:729] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee7950c919048da8f77a00266eccf8a, State: Initialized, Role: FOLLOWER
I20260925 16:39:56.504443 12096 consensus_queue.cc:260] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.505805 12096 ts_tablet_manager.cc:1428] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:39:56.506193 11802 heartbeater.cc:503] Master 127.0.97.254:45949 was elected leader, sending a full tablet report...
W20260925 16:39:56.529651 11940 tablet.cc:2369] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:39:56.627996 12074 tablet.cc:2369] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:39:56.654593 11803 tablet.cc:2369] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:39:56.669879 12100 raft_consensus.cc:482] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:39:56.670083 12100 raft_consensus.cc:504] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.671401 12100 leader_election.cc:288] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c905443f5840b1a7e014b6e71cfcdb (127.0.97.195:39537), b4a6eb4617df4f4faec088548081a53b (127.0.97.194:41961)
I20260925 16:39:56.678115 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.672908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:36059 (local address 127.0.97.194:41961)
0925 16:39:56.673045 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.673050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.673356 (+ 306us) server_negotiation.cc:407] Connection header received
0925 16:39:56.673571 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.673574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.673629 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.673685 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:56.674114 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.674920 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.676642 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.677343 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.677444 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.677501 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.677869 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.677952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:39:56.678890 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.672513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41961 (local address 127.0.97.193:36059)
0925 16:39:56.673236 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.673251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.673436 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.673840 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.673843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.673862 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:56.673965 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.673985 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.675096 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.675100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.676482 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.676494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.677615 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.677618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.677720 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.677745 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.678547 (+ 802us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.678714 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":65,"thread_start_us":99,"threads_started":1}
I20260925 16:39:56.679759 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28541e5485c646df8329c9bc53980a00" candidate_uuid: "6ee7950c919048da8f77a00266eccf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a6eb4617df4f4faec088548081a53b" is_pre_election: true
I20260925 16:39:56.680085 11875 raft_consensus.cc:2454] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee7950c919048da8f77a00266eccf8a in term 0.
I20260925 16:39:56.680646 11691 leader_election.cc:302] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee7950c919048da8f77a00266eccf8a, b4a6eb4617df4f4faec088548081a53b; no voters:
I20260925 16:39:56.680980 12100 raft_consensus.cc:2781] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:39:56.681061 12100 raft_consensus.cc:482] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:39:56.681113 12100 raft_consensus.cc:3037] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:56.682224 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.672049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:53471 (local address 127.0.97.195:39537)
0925 16:39:56.672185 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.672190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.672208 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:39:56.673299 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.673303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.673358 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.673409 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:39:56.677652 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.678401 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.680895 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.681562 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.681654 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.681701 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.682017 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.682094 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:39:56.682619 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.671656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39537 (local address 127.0.97.193:53471)
0925 16:39:56.671991 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.672007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:39:56.672441 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:39:56.673553 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:39:56.673555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:39:56.673573 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:39:56.677505 (+ 3932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.677516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.678563 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.678566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.680754 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:39:56.680764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.681791 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.681795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:39:56.681879 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.681901 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:39:56.682398 (+ 497us) client_negotiation.cc:240] Negotiation successful
0925 16:39:56.682472 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":13,"thread_start_us":135,"threads_started":1}
I20260925 16:39:56.682629 12100 raft_consensus.cc:504] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.683353 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28541e5485c646df8329c9bc53980a00" candidate_uuid: "6ee7950c919048da8f77a00266eccf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" is_pre_election: true
I20260925 16:39:56.683539 12100 leader_election.cc:288] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [CANDIDATE]: Term 1 election: Requested vote from peers a8c905443f5840b1a7e014b6e71cfcdb (127.0.97.195:39537), b4a6eb4617df4f4faec088548081a53b (127.0.97.194:41961)
I20260925 16:39:56.683617 12026 raft_consensus.cc:2454] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee7950c919048da8f77a00266eccf8a in term 0.
I20260925 16:39:56.683984 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28541e5485c646df8329c9bc53980a00" candidate_uuid: "6ee7950c919048da8f77a00266eccf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c905443f5840b1a7e014b6e71cfcdb"
I20260925 16:39:56.684078 12026 raft_consensus.cc:3037] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:56.684737 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28541e5485c646df8329c9bc53980a00" candidate_uuid: "6ee7950c919048da8f77a00266eccf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a6eb4617df4f4faec088548081a53b"
I20260925 16:39:56.684873 11875 raft_consensus.cc:3037] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:39:56.685307 12026 raft_consensus.cc:2454] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee7950c919048da8f77a00266eccf8a in term 1.
I20260925 16:39:56.686342 11875 raft_consensus.cc:2454] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee7950c919048da8f77a00266eccf8a in term 1.
I20260925 16:39:56.686743 11691 leader_election.cc:302] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee7950c919048da8f77a00266eccf8a, b4a6eb4617df4f4faec088548081a53b; no voters:
I20260925 16:39:56.686967 12100 raft_consensus.cc:2781] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:39:56.687225 12100 raft_consensus.cc:686] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [term 1 LEADER]: Becoming Leader. State: Replica: 6ee7950c919048da8f77a00266eccf8a, State: Running, Role: LEADER
I20260925 16:39:56.687597 12100 consensus_queue.cc:237] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } }
I20260925 16:39:56.691069 11589 catalog_manager.cc:5495] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ee7950c919048da8f77a00266eccf8a (127.0.97.193). New cstate: current_term: 1 leader_uuid: "6ee7950c919048da8f77a00266eccf8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee7950c919048da8f77a00266eccf8a" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 33161 } health_report { overall_health: HEALTHY } } }
I20260925 16:39:56.733652 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.729669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.0.97.193:33161)
0925 16:39:56.729823 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:39:56.729828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:39:56.729846 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:39:56.729945 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:39:56.729948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:39:56.730038 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:39:56.730164 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:39:56.730426 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.731001 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.731951 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:39:56.732198 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:39:56.732315 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:39:56.732661 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:39:56.733169 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:39:56.733335 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 16:39:56.733421 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:39:56.733479 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:39:56.739919 12026 raft_consensus.cc:1261] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 1 FOLLOWER]: Refusing update from remote peer 6ee7950c919048da8f77a00266eccf8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:39:56.739944 11875 raft_consensus.cc:1261] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 1 FOLLOWER]: Refusing update from remote peer 6ee7950c919048da8f77a00266eccf8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:39:56.740756 12104 consensus_queue.cc:1035] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c905443f5840b1a7e014b6e71cfcdb" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:39:56.741240 12100 consensus_queue.cc:1035] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a6eb4617df4f4faec088548081a53b" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:39:56.751507 12115 mvcc.cc:205] Tried to move back new op lower bound from 7333291609038954496 to 7333291608835874816. Current Snapshot: MvccSnapshot[applied={T|T < 7333291609038954496}]
I20260925 16:39:56.761133 11587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:39:56.761343 11587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:39:56.763765 11587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ebe51d2e17a4f6b9f04546b97436a1c: Sending DeleteTablet for 3 replicas of tablet 28541e5485c646df8329c9bc53980a00
I20260925 16:39:56.764532 11855 tablet_service.cc:1514] Processing DeleteTablet for tablet 28541e5485c646df8329c9bc53980a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:56 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:37636
I20260925 16:39:56.764617 11995 tablet_service.cc:1514] Processing DeleteTablet for tablet 28541e5485c646df8329c9bc53980a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:56 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:43236
I20260925 16:39:56.765058 12124 tablet_replica.cc:330] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: stopping tablet replica
I20260925 16:39:56.765038 11734 tablet_service.cc:1514] Processing DeleteTablet for tablet 28541e5485c646df8329c9bc53980a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:39:56 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:34844
I20260925 16:39:56.765360 12123 tablet_replica.cc:330] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: stopping tablet replica
I20260925 16:39:56.765491 12125 tablet_replica.cc:330] T 28541e5485c646df8329c9bc53980a00 P 6ee7950c919048da8f77a00266eccf8a: stopping tablet replica
I20260925 16:39:56.765669 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 11670
I20260925 16:39:56.766323 12123 raft_consensus.cc:2229] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:39:56.766610 12123 raft_consensus.cc:2258] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:39:56.766830 12124 raft_consensus.cc:2229] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:39:56.767081 12124 raft_consensus.cc:2258] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:39:56.770298 12124 ts_tablet_manager.cc:1905] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:39:56.772471 12124 ts_tablet_manager.cc:1918] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:39:56.772547 12124 log.cc:1196] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/28541e5485c646df8329c9bc53980a00
I20260925 16:39:56.772837 12124 ts_tablet_manager.cc:1939] T 28541e5485c646df8329c9bc53980a00 P b4a6eb4617df4f4faec088548081a53b: Deleting consensus metadata
I20260925 16:39:56.773660 12123 ts_tablet_manager.cc:1905] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:39:56.773980 11576 catalog_manager.cc:4841] TS b4a6eb4617df4f4faec088548081a53b (127.0.97.194:41961): tablet 28541e5485c646df8329c9bc53980a00 (table test-table [id=15cce40afe994aa08afe5c5f224108ec]) successfully deleted
W20260925 16:39:56.774416 11573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:33161 (error 108)
I20260925 16:39:56.774672 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 11809
I20260925 16:39:56.777150 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:39:56.776882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:33161 (local address 127.0.0.1:34862)
0925 16:39:56.777008 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 16:39:56.777074 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:33161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:39:56.777324 11573 catalog_manager.cc:4584] TS 6ee7950c919048da8f77a00266eccf8a (127.0.97.193:33161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28541e5485c646df8329c9bc53980a00: Network error: Client connection negotiation failed: client connection to 127.0.97.193:33161: connect: Connection refused (error 111)
I20260925 16:39:56.777354 12123 ts_tablet_manager.cc:1918] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:39:56.777447 12123 log.cc:1196] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/28541e5485c646df8329c9bc53980a00
I20260925 16:39:56.777791 12123 ts_tablet_manager.cc:1939] T 28541e5485c646df8329c9bc53980a00 P a8c905443f5840b1a7e014b6e71cfcdb: Deleting consensus metadata
I20260925 16:39:56.779011 11573 catalog_manager.cc:4841] TS a8c905443f5840b1a7e014b6e71cfcdb (127.0.97.195:39537): tablet 28541e5485c646df8329c9bc53980a00 (table test-table [id=15cce40afe994aa08afe5c5f224108ec]) successfully deleted
I20260925 16:39:56.780984 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 11944
I20260925 16:39:56.786583 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 11557
I20260925 16:39:56.792821 391 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:39:57.127399 391 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:39:57.127516 391 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:39:57.472688 391 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:39:57.472 UTC [11143] LOG: received smart shutdown request
2026-09-25 16:39:57.476 UTC [11143] LOG: background worker "logical replication launcher" (PID 11152) exited with exit code 1
2026-09-25 16:39:57.476 UTC [11147] LOG: shutting down
2026-09-25 16:39:57.648 UTC [11143] LOG: database system is shut down
2026-09-25T16:39:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46870 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:39:57 dist-test-slave-kxl8 krb5kdc[12135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:39:57 dist-test-slave-kxl8 krb5kdc[12135](info): set up 2 sockets
Sep 25 16:39:57 dist-test-slave-kxl8 krb5kdc[12135](info): commencing operation
krb5kdc: starting...
W20260925 16:39:58.813208 391 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:39:58 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:39:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangeradmin_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangeradmin_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangerlookup_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangerlookup_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
I20260925 16:39:58.978106 391 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVEDvT3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/postgres -l logfile start
2026-09-25 16:40:01.518 UTC [12165] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:40:01.518 UTC [12165] LOG: listening on IPv4 address "127.0.97.212", port 49321
2026-09-25 16:40:01.523 UTC [12165] LOG: listening on Unix socket "/tmp/.s.PGSQL.49321"
2026-09-25 16:40:01.558 UTC [12168] LOG: database system was shut down at 2026-09-25 16:40:00 UTC
2026-09-25 16:40:01.562 UTC [12165] LOG: database system is ready to accept connections
I20260925 16:40:02.506181 391 mini_postgres.cc:96] Postgres bound to 49321
2026-09-25 16:40:02.513 UTC [12176] FATAL: database "slave" does not exist
127.0.97.212:49321 - accepting connections
I20260925 16:40:02.514159 391 mini_ranger.cc:161] Starting Ranger...
I20260925 16:40:02.531144 391 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:40:03.024775 391 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:40:03.024857 391 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin
2026-09-25 16:40:03,306 [I] DB FLAVOR :POSTGRES
2026-09-25 16:40:03,307 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:40:03,307 [I] Checking connection..
2026-09-25 16:40:03,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:40:03,547 [I] Checking connection passed.
2026-09-25 16:40:03,547 [I] --------- Verifying version history table ---------
2026-09-25 16:40:03,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:40:03,813 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:40:03,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:40:04,080 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:40:04,080 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:40:04,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:40:04.332 UTC [12240] WARNING: there is no transaction in progress
2026-09-25 16:40:04,338 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:40:04,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:40:04,603 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:40:04,603 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:40:04,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:40:04,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:40:05,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kxl8.c.gcp-upstream.internal','N') ;"
2026-09-25 16:40:05,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:40:05,581 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:40:05,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:40:05,826 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:40:05,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:40:06,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:40:06,342 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:40:06,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:40:07.343 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.361 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.378 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.393 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.410 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.429 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.487 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.494 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.509 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.520 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.531 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.546 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.557 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.564 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.570 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:07.578 UTC [12348] WARNING: there is no transaction in progress
2026-09-25 16:40:08,242 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:40:08,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:40:08,494 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:40:08,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:40:08,750 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:40:08,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:40:08,994 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:40:08,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:40:09,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:40:09,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kxl8.c.gcp-upstream.internal';"
2026-09-25 16:40:09,716 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:40:09,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:40:09,942 [I] DB_PATCHES have already been applied
I20260925 16:40:09.948376 391 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVEDvT3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:40:09.948447 391 mini_ranger.cc:193] Using host: 127.0.97.212
I20260925 16:40:09.951853 391 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.212:52231
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:40:11.061648 391 mini_ranger.cc:160] Time spent starting Ranger: real 8.547s user 0.005s sys 0.005s
Sep 25, 2026 4:40:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:40:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 4:40:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:40:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:40:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:40:32.811444 391 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangerkms_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/rangerkms_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/HTTP_127.0.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:40:32 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354432, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:40:32.897418 391 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:40:32.906889 391 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:40:33.794786 391 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:40:33.794872 391 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms
I20260925 16:40:33.794895 391 mini_ranger_kms.cc:227] Using postgres at 127.0.97.212:49321
2026-09-25 16:40:33,959 [I] DB FLAVOR :POSTGRES
2026-09-25 16:40:33,959 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:40:33,959 [I] Checking connection
2026-09-25 16:40:33,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:40:34,221 [I] connection success
2026-09-25 16:40:34,221 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:40:34,221 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:40:34,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:40:34,487 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:40:34,487 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:40:34,487 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:40:34,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.212:49321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:40:34,752 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:40:35.225669 391 mini_ranger_kms.cc:310] Created kms service
I20260925 16:40:35.424718 391 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:40:35.466814 391 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:40:35 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354432, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:35.848215 391 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:40:35.848462 391 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVEDvT3/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVEDvT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:40:35.848484 391 mini_ranger_kms.cc:240] Using host: 127.0.97.212
I20260925 16:40:35.852197 391 mini_ranger_kms.cc:276] Ranger KMS PID: 12528
I20260925 16:40:35.852284 391 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.212:58399
Sep 25, 2026 4:40:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:40:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:40:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:40:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:40:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:40:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 16:40:38.129154 391 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.232s user 0.002s sys 0.147s
I20260925 16:40:38.129304 391 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:40:38.129372 391 mini_ranger_kms.cc:401] 127.0.97.212:58399/kms/v1/keys
Sep 25 16:40:38 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 16:40:38 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
Sep 25 16:40:39 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.212@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
Sep 25 16:40:41 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:40:41.337589 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38039
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:38039
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:41.445242 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:41.445566 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:41.445637 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:41.449589 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:41.449679 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:41.449717 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:41.449774 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:41.449813 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:41.455183 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.212:58399/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:38039
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:41.456281 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:41.457326 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:41.466223 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:41.466182 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:41.466365 12579 server_base.cc:1034] running on GCE node
W20260925 16:40:41.466275 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:41.466933 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:41.468034 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:41.469228 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790354441469202 us; error 50 us; skew 500 ppm
Sep 25 16:40:41 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:41.476653 12579 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:41.477753 12579 webserver.cc:460] Webserver started at http://127.0.97.254:43223/ using document root <none> and password file <none>
I20260925 16:40:41.478322 12579 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:41.478402 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:41.478627 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:40:41 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:41.540712 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "83d386b9114d4fa981c7cd1daab344e0"
format_stamp: "Formatted at 2026-09-25 16:40:41 on dist-test-slave-kxl8"
server_key: "3db747c180b3e77dd9cfad5351fcdfea"
server_key_iv: "532441b661c8d857dae9e71588db1b36"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:41.541498 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "83d386b9114d4fa981c7cd1daab344e0"
format_stamp: "Formatted at 2026-09-25 16:40:41 on dist-test-slave-kxl8"
server_key: "3db747c180b3e77dd9cfad5351fcdfea"
server_key_iv: "532441b661c8d857dae9e71588db1b36"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:41.545585 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:40:41.562682 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:41.564391 12579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:41.564527 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "83d386b9114d4fa981c7cd1daab344e0"
format_stamp: "Formatted at 2026-09-25 16:40:41 on dist-test-slave-kxl8"
server_key: "3db747c180b3e77dd9cfad5351fcdfea"
server_key_iv: "532441b661c8d857dae9e71588db1b36"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:41.564663 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:41.584429 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:41.587720 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:41.587921 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:41.596509 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:38039 every 8 connection(s)
I20260925 16:40:41.596513 12579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:38039
I20260925 16:40:41.597805 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:41.600975 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:41.605075 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 12579
I20260925 16:40:41.605201 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:40:41.605544 391 external_mini_cluster.cc:1428] Setting key 179d6debaa99cd57f3e587797bd6f5c0
I20260925 16:40:41.608186 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Bootstrap starting.
I20260925 16:40:41.610736 12647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:41.611572 12647 log.cc:824] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:41.613806 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: No bootstrap required, opened a new log
Sep 25 16:40:41 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:41.617347 12647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:41.617574 12647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:41.617660 12647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d386b9114d4fa981c7cd1daab344e0, State: Initialized, Role: FOLLOWER
I20260925 16:40:41.618110 12647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:41.618252 12647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:41.618336 12647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:41.618432 12647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:41.619225 12647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:41.619551 12647 leader_election.cc:302] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d386b9114d4fa981c7cd1daab344e0; no voters:
I20260925 16:40:41.619874 12647 leader_election.cc:288] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:40:41.619935 12652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:41.620141 12652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 LEADER]: Becoming Leader. State: Replica: 83d386b9114d4fa981c7cd1daab344e0, State: Running, Role: LEADER
I20260925 16:40:41.620483 12652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:41.620584 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:41.607042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.97.254:38039)
0925 16:40:41.607491 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:41.607507 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:41.607706 (+ 199us) server_negotiation.cc:407] Connection header received
0925 16:40:41.608475 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:41.608493 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:41.608841 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:41.609168 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:41.610008 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:41.611095 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:41.612218 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:41.612779 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:41.612907 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:41.615659 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:41.615663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:41.615676 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:41.615702 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:41.617432 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:41.617946 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:41.617951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:41.617964 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:41.618067 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:41.618394 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:41.618397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:41.618399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:41.618814 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:41.619020 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:41.619303 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 16:40:41.619606 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260925 16:40:41.621088 12647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:41.623049 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d386b9114d4fa981c7cd1daab344e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } } }
I20260925 16:40:41.623164 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:41.623289 12647 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:40:41.623514 12666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:41.624347 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d386b9114d4fa981c7cd1daab344e0. Latest consensus state: current_term: 1 leader_uuid: "83d386b9114d4fa981c7cd1daab344e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } } }
I20260925 16:40:41.624436 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:41.628870 12666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:41.634339 12666 catalog_manager.cc:1347] Generated new cluster ID: 5e3ad3e15f32440daec0a9df0d5243a2
I20260925 16:40:41.634397 12666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:41.656009 12666 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:40:41.657481 12666 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:41.664709 12666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Generated new TSK 0
I20260925 16:40:41.665616 12666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:40:42.316427 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:42.444337 12692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:42.444643 12692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:42.444715 12692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:42.448935 12692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:42.449025 12692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:42.449117 12692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:40:42.454805 12692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:42.455885 12692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:42.456908 12692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:40:42.468991 12692 server_base.cc:1034] running on GCE node
W20260925 16:40:42.468917 12701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:42.468850 12699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:42.471625 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:42.472164 12692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:42.472927 12692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:42.474154 12692 hybrid_clock.cc:648] HybridClock initialized: now 1790354442474082 us; error 58 us; skew 500 ppm
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:42.477005 12692 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:40:42.478112 12692 webserver.cc:460] Webserver started at http://127.0.97.193:44223/ using document root <none> and password file <none>
I20260925 16:40:42.478780 12692 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:42.478963 12692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:42.479264 12692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:42.494243 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "3aeccf5966eb47a1a83da5389cc68286"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "d5f270cf56dae0e647f9dadc1aa973df"
server_key_iv: "c73b0966d283ff09021dd7b66dcadb8a"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.495195 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "3aeccf5966eb47a1a83da5389cc68286"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "d5f270cf56dae0e647f9dadc1aa973df"
server_key_iv: "c73b0966d283ff09021dd7b66dcadb8a"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.502887 12692 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260925 16:40:42.524118 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.525743 12692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:40:42.525899 12692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "3aeccf5966eb47a1a83da5389cc68286"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "d5f270cf56dae0e647f9dadc1aa973df"
server_key_iv: "c73b0966d283ff09021dd7b66dcadb8a"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.526049 12692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:42.567822 12692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:42.584364 12692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:42.584620 12692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:42.585261 12692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:42.586404 12692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:42.586525 12692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.586650 12692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:42.586736 12692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.607246 12692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:34019
I20260925 16:40:42.611822 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:34019 every 8 connection(s)
I20260925 16:40:42.612802 12692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:40:42.617179 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 12692
I20260925 16:40:42.617334 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:40:42.617664 391 external_mini_cluster.cc:1428] Setting key ffd85ae57cf0cacc6dd3f0f6308359f5
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:42.629328 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:42.614946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:32977 (local address 127.0.97.254:38039)
0925 16:40:42.615234 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:42.615243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:42.616001 (+ 758us) server_negotiation.cc:407] Connection header received
0925 16:40:42.616968 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:42.616972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:42.617029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:42.617144 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:42.618422 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.619053 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.620809 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.621219 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.621321 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:42.624396 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:42.624402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:42.624408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:42.624441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:42.626488 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:42.627180 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:42.627185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:42.627190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:42.627284 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:42.627814 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:42.627819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:42.627820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:42.628086 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:42.628190 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:42.628980 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 16:40:42.629116 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:40:42.631078 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:42.615405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.193:32977)
0925 16:40:42.615848 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:42.615901 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:42.616761 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:42.617354 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:42.617363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:42.617664 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:42.618286 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:42.618304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.619204 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.619207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:42.620662 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:42.620677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.621539 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.621545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:42.621759 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:42.622486 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:42.622502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:42.624203 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:42.626646 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:42.626655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:42.626675 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:42.627026 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:42.627404 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:42.627410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:42.627415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:42.627691 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:42.628303 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:42.628309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:42.628668 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 16:40:42.629827 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0925 16:40:42.630091 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260925 16:40:42.632402 12825 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:40:42.632645 12825 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:42.633231 12825 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:42.634953 12611 ts_manager.cc:194] Registered new tserver with Master: 3aeccf5966eb47a1a83da5389cc68286 (127.0.97.193:34019)
I20260925 16:40:42.636098 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:32977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:40:42.653661 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 16:40:42.781982 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:42.782280 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:42.782410 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:42.786818 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:42.786901 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:42.787006 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:40:42.792909 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:42.794229 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:42.795334 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:40:42.807256 12831 server_base.cc:1034] running on GCE node
W20260925 16:40:42.807492 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:42.807660 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:42.807507 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:42.808288 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:42.809063 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:42.810258 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790354442810239 us; error 77 us; skew 500 ppm
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:42.814085 12831 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:40:42.815301 12831 webserver.cc:460] Webserver started at http://127.0.97.194:38013/ using document root <none> and password file <none>
I20260925 16:40:42.816074 12831 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:42.816236 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:42.816510 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:42.834616 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "5965656ba0514a2fa850b92511b25dc5"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "7f2a57694cfa70c93f87e684c0a22018"
server_key_iv: "0a7f916b4d99669bc9734aa763a16974"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.835446 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "5965656ba0514a2fa850b92511b25dc5"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "7f2a57694cfa70c93f87e684c0a22018"
server_key_iv: "0a7f916b4d99669bc9734aa763a16974"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.845825 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260925 16:40:42.865687 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.867468 12831 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260925 16:40:42.867594 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "5965656ba0514a2fa850b92511b25dc5"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "7f2a57694cfa70c93f87e684c0a22018"
server_key_iv: "0a7f916b4d99669bc9734aa763a16974"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:42.867738 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:42.892935 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:42.931443 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:42.931730 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:42.932677 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:42.934192 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:42.934279 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.934374 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:42.934427 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:42.954308 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:44171
I20260925 16:40:42.955603 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:40:42.958158 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:44171 every 8 connection(s)
I20260925 16:40:42.962312 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 12831
I20260925 16:40:42.962450 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:40:42.962798 391 external_mini_cluster.cc:1428] Setting key 55007d4366d05ae315adccaeea880a32
Sep 25 16:40:42 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:40:42.978343 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:42.963122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:35407 (local address 127.0.97.254:38039)
0925 16:40:42.963314 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:42.963318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:42.963336 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:40:42.964008 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:42.964012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:42.964067 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:42.964121 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:42.965968 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.966742 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.969563 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.970179 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.970289 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:42.973937 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:42.973941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:42.973944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:42.973971 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:42.975885 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:42.976495 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:42.976498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:42.976500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:42.976584 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:42.976993 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:42.976997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:42.976998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:42.977215 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:42.977321 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:42.977996 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 16:40:42.978170 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 16:40:42.980010 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:42.961967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.194:35407)
0925 16:40:42.962687 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:42.962740 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:42.963758 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:42.964526 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:42.964535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:42.964896 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:42.965802 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:42.965822 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.967081 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.967085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:42.969353 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:42.969387 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:42.970500 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:42.970503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:42.970765 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:42.971679 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:42.971700 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:42.973733 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:42.976016 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:42.976022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:42.976039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:42.976377 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:42.976701 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:42.976704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:42.976706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:42.976886 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:42.977386 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:42.977392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:42.977719 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 16:40:42.978514 (+ 795us) client_negotiation.cc:240] Negotiation successful
0925 16:40:42.978870 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":170,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:40:42.981426 12961 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
I20260925 16:40:42.981680 12961 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:42.982245 12961 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:42.983330 12611 ts_manager.cc:194] Registered new tserver with Master: 5965656ba0514a2fa850b92511b25dc5 (127.0.97.194:44171)
I20260925 16:40:42.984071 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:35407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:40:43.009568 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:43.154799 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:43.155184 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:43.155308 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
W20260925 16:40:43.159798 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:43.159881 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:43.159976 12966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:40:43.167903 12966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:43.169503 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:43.170921 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:43.182952 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.183544 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.183612 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:43.184798 12966 server_base.cc:1034] running on GCE node
I20260925 16:40:43.185371 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:43.186139 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:43.187347 12966 hybrid_clock.cc:648] HybridClock initialized: now 1790354443187309 us; error 41 us; skew 500 ppm
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:43.198609 12966 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:40:43.199918 12966 webserver.cc:460] Webserver started at http://127.0.97.195:46037/ using document root <none> and password file <none>
I20260925 16:40:43.200703 12966 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:43.200842 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:43.201131 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:43.219774 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82"
format_stamp: "Formatted at 2026-09-25 16:40:43 on dist-test-slave-kxl8"
server_key: "16ff5a84d5a61c8b9ad25079c21008e7"
server_key_iv: "53394b4477a8394d0c00857d5ec18912"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:43.220875 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82"
format_stamp: "Formatted at 2026-09-25 16:40:43 on dist-test-slave-kxl8"
server_key: "16ff5a84d5a61c8b9ad25079c21008e7"
server_key_iv: "53394b4477a8394d0c00857d5ec18912"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:43.237534 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260925 16:40:43.269258 12981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.271127 12966 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260925 16:40:43.271251 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82"
format_stamp: "Formatted at 2026-09-25 16:40:43 on dist-test-slave-kxl8"
server_key: "16ff5a84d5a61c8b9ad25079c21008e7"
server_key_iv: "53394b4477a8394d0c00857d5ec18912"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:43.271379 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:43.309067 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:43.313297 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:43.313522 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:43.314241 12966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:43.315267 12966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:43.315336 12966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.315444 12966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:43.315487 12966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.326535 12966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37803
I20260925 16:40:43.326581 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37803 every 8 connection(s)
I20260925 16:40:43.327793 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:40:43.337494 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 12966
I20260925 16:40:43.337635 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:40:43.337936 391 external_mini_cluster.cc:1428] Setting key 3cd570aeff8c36a1b0f87a53e83a22cd
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:43.344059 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:43.329765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:51747 (local address 127.0.97.254:38039)
0925 16:40:43.329898 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:43.329903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:43.330997 (+ 1094us) server_negotiation.cc:407] Connection header received
0925 16:40:43.332600 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:43.332603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:43.332657 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:43.332758 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:43.334106 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.334953 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.336575 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.336925 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.337044 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:43.339897 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:43.339901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:43.339904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:43.339930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:43.341743 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:43.342387 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:43.342390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:43.342392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:43.342477 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:43.342880 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:43.342883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:43.342885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:43.343097 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:43.343186 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:43.343776 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 16:40:43.343900 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:40:43.345185 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:43.330271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.195:51747)
0925 16:40:43.330845 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:43.330899 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:43.332340 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:43.332944 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:43.332953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:43.333326 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:43.333944 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:43.333960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.335089 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.335092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:43.336438 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:43.336448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.337086 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.337090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:43.337331 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:43.338025 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:43.338054 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:43.339716 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:43.341870 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:43.341876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:43.341892 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:43.342282 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:43.342588 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:43.342592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:43.342594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:43.342772 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:43.343204 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:43.343211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:43.343489 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 16:40:43.343713 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 16:40:43.344037 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260925 16:40:43.346477 13095 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
I20260925 16:40:43.346740 13095 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:43.347309 13095 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:43.348472 12611 ts_manager.cc:194] Registered new tserver with Master: 6f9e365a20d64b6db5a0c8d8dddb9e82 (127.0.97.195:37803)
I20260925 16:40:43.349189 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:51747
I20260925 16:40:43.353075 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:40:43.353792 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 12692
I20260925 16:40:43.361265 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 12831
I20260925 16:40:43.367887 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 12966
I20260925 16:40:43.374598 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 12579
I20260925 16:40:43.380962 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38039
--webserver_interface=127.0.97.254
--webserver_port=43223
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:38039
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:43.488950 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:43.489210 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:43.489286 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:43.493458 13098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:43.493512 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:43.493531 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:43.493589 13098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:43.493641 13098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:43.499128 13098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:38039
--ranger_config_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:43.500155 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:43.501155 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:43.509796 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.509790 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.509790 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:43.510094 13098 server_base.cc:1034] running on GCE node
I20260925 16:40:43.510656 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:43.511749 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:43.512931 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790354443512902 us; error 47 us; skew 500 ppm
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:43.515578 13098 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:43.516424 13098 webserver.cc:460] Webserver started at http://127.0.97.254:43223/ using document root <none> and password file <none>
I20260925 16:40:43.516968 13098 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:43.517033 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:43.585645 13098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:43.587914 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.589190 13098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:43.589357 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "83d386b9114d4fa981c7cd1daab344e0"
format_stamp: "Formatted at 2026-09-25 16:40:41 on dist-test-slave-kxl8"
server_key: "3db747c180b3e77dd9cfad5351fcdfea"
server_key_iv: "532441b661c8d857dae9e71588db1b36"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:43.589920 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:43.615747 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:43.618460 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:43.618654 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:43.625804 13098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:38039
I20260925 16:40:43.625830 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:38039 every 8 connection(s)
I20260925 16:40:43.626806 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:43.628222 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13098
I20260925 16:40:43.628643 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:34019
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:38039
--webserver_port=44223
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:43.631228 13167 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:40:43.632808 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Bootstrap starting.
I20260925 16:40:43.649139 13167 log.cc:824] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:43.655794 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:40:43.656215 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Bootstrap complete.
I20260925 16:40:43.660486 13167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:43.660907 13167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d386b9114d4fa981c7cd1daab344e0, State: Initialized, Role: FOLLOWER
I20260925 16:40:43.661453 13167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:43.661610 13167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:43.661692 13167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:43.661823 13167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:40:43.663822 13167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:43.664160 13167 leader_election.cc:302] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d386b9114d4fa981c7cd1daab344e0; no voters:
I20260925 16:40:43.664501 13167 leader_election.cc:288] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:40:43.664588 13171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:40:43.664832 13171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [term 2 LEADER]: Becoming Leader. State: Replica: 83d386b9114d4fa981c7cd1daab344e0, State: Running, Role: LEADER
I20260925 16:40:43.665194 13171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } }
I20260925 16:40:43.665838 13167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:43.666654 13172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83d386b9114d4fa981c7cd1daab344e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } } }
I20260925 16:40:43.666795 13172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:43.667127 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d386b9114d4fa981c7cd1daab344e0. Latest consensus state: current_term: 2 leader_uuid: "83d386b9114d4fa981c7cd1daab344e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d386b9114d4fa981c7cd1daab344e0" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38039 } } }
I20260925 16:40:43.667249 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:43.667802 13179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:43.668325 13167 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:40:43.673969 13179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:43.674702 13179 catalog_manager.cc:1259] Loaded cluster ID: 5e3ad3e15f32440daec0a9df0d5243a2
I20260925 16:40:43.674809 13179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:43.677523 13179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:43.678260 13179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Loaded TSK: 0
I20260925 16:40:43.679577 13179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:40:43.775705 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:43.776146 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:43.776268 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:43.782527 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:43.782618 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:43.782707 13169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:40:43.790750 13169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=44223
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:43.792125 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:43.793553 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:43.806666 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.809320 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:43.809685 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:43.814842 13169 server_base.cc:1034] running on GCE node
I20260925 16:40:43.815446 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:43.816293 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:43.817495 13169 hybrid_clock.cc:648] HybridClock initialized: now 1790354443817426 us; error 91 us; skew 500 ppm
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:43.821025 13169 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:40:43.822146 13169 webserver.cc:460] Webserver started at http://127.0.97.193:44223/ using document root <none> and password file <none>
I20260925 16:40:43.822860 13169 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:43.822952 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:43.839975 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:40:43.842775 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.844112 13169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 16:40:43.844249 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "3aeccf5966eb47a1a83da5389cc68286"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "d5f270cf56dae0e647f9dadc1aa973df"
server_key_iv: "c73b0966d283ff09021dd7b66dcadb8a"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:43.844808 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:43.866518 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:43.871268 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:43.871452 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:43.872140 13169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:43.873189 13169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:43.873257 13169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.873333 13169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:43.873363 13169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:43.884243 13169 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:34019
I20260925 16:40:43.885003 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:34019 every 8 connection(s)
I20260925 16:40:43.885506 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:40:43.887703 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13169
I20260925 16:40:43.888193 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:44171
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:38039
--webserver_port=38013
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:40:43 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:43.927873 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:43.897327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.193:36585)
0925 16:40:43.899024 (+ 1697us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:43.899067 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:43.899834 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:43.901518 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:43.901526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:43.901874 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:43.902348 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:43.902363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.912079 (+ 9716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.912084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:43.913612 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:43.913626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.919534 (+ 5908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.919537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:43.919777 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:43.920543 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:43.920565 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:43.922206 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:43.924688 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:43.924694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:43.924707 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:43.925020 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:43.925441 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:43.925444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:43.925446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:43.925582 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:43.926171 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:43.926176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:43.926404 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 16:40:43.926601 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 16:40:43.926933 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1504,"thread_start_us":150,"threads_started":1}
I20260925 16:40:43.927862 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:43.897104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:36585 (local address 127.0.97.254:38039)
0925 16:40:43.900236 (+ 3132us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:43.900254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:43.900288 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:40:43.900510 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:43.900521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:43.901066 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:43.901458 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:43.907936 (+ 6478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.911877 (+ 3941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.913787 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:43.914968 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:43.919320 (+ 4352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:43.922445 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:43.922452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:43.922468 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:43.922502 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:43.924551 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:43.925174 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:43.925180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:43.925185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:43.925345 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:43.925686 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:43.925690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:43.925691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:43.926040 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:43.926183 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:43.926725 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 16:40:43.926960 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2981,"thread_start_us":150,"threads_started":1}
I20260925 16:40:43.930672 13329 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
I20260925 16:40:43.930940 13329 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:43.931448 13329 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:43.933398 13329 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:38039 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:40:43.933516 13329 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:43.933689 13329 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:43.934532 13329 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:43.934724 13329 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:43.935451 13329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:40:44.051067 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:44.051455 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:44.051532 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:44.057849 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:44.057936 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:44.058038 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:40:44.066179 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=38013
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:44.067672 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:44.069075 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:44.080753 13339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:44.080823 13331 server_base.cc:1034] running on GCE node
W20260925 16:40:44.080762 13338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:44.080749 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:44.081498 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:44.082289 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:44.083463 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790354444083447 us; error 31 us; skew 500 ppm
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:44.086324 13331 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:40:44.087354 13331 webserver.cc:460] Webserver started at http://127.0.97.194:38013/ using document root <none> and password file <none>
I20260925 16:40:44.088027 13331 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:44.088112 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:44.114274 13331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:40:44.117126 13349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:44.118556 13331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:44.118695 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "5965656ba0514a2fa850b92511b25dc5"
format_stamp: "Formatted at 2026-09-25 16:40:42 on dist-test-slave-kxl8"
server_key: "7f2a57694cfa70c93f87e684c0a22018"
server_key_iv: "0a7f916b4d99669bc9734aa763a16974"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:44.119278 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:44.139835 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:44.143168 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:44.143484 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:44.144198 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:44.145131 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:44.145203 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:44.145295 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:44.145358 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 16:40:44.157261 13331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:44171
I20260925 16:40:44.157310 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:44171 every 8 connection(s)
I20260925 16:40:44.159119 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:40:44.166534 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13331
I20260925 16:40:44.167132 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.212:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:37803
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:38039
--webserver_port=46037
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:44.175778 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:44.162454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37635 (local address 127.0.97.254:38039)
0925 16:40:44.162606 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:44.162610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:44.162628 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:40:44.163191 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:44.163195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:44.163253 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:44.163354 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:44.164522 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.165085 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.166509 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.166866 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.166954 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:44.170707 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:44.170714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:44.170720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:44.170755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:44.172773 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.173561 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.173577 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.173581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.173676 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.174217 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.174221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.174223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.174457 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:44.174541 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:44.175356 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 16:40:44.175532 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:40:44.177965 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:44.161460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.194:37635)
0925 16:40:44.161961 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:44.162014 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:44.162956 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:44.163542 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:44.163550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:44.163825 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:44.164346 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:44.164364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.165218 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.165221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:44.166360 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:44.166372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.167523 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.167530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:44.167781 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:44.168534 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:44.168560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:44.170496 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:44.172942 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:44.172950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:44.172972 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:44.173428 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:44.173793 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:44.173796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:44.173797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:44.173979 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:44.174616 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:44.174624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:44.175004 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 16:40:44.176326 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0925 16:40:44.176742 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260925 16:40:44.179736 13463 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
I20260925 16:40:44.179998 13463 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.180611 13463 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:44.182449 13463 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:38039 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:40:44.182582 13463 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.182775 13463 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:44.183753 13463 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.183945 13463 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:44.184855 13463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:40:44.312529 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:44.312829 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:44.312911 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:44.317234 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:44.317348 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:44.317451 13466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:40:44.323985 13466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.212:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:37803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=46037
--tserver_master_addrs=127.0.97.254:38039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:44.325250 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:44.326330 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:44.337460 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:44.337862 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:44.337471 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:44.341490 13466 server_base.cc:1034] running on GCE node
I20260925 16:40:44.341914 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:44.342612 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:44.343786 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790354444343768 us; error 32 us; skew 500 ppm
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:44.347587 13466 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:40:44.348794 13466 webserver.cc:460] Webserver started at http://127.0.97.195:46037/ using document root <none> and password file <none>
I20260925 16:40:44.349644 13466 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:44.349738 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:44.368016 13466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:40:44.370872 13483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:44.372521 13466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:44.372661 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82"
format_stamp: "Formatted at 2026-09-25 16:40:43 on dist-test-slave-kxl8"
server_key: "16ff5a84d5a61c8b9ad25079c21008e7"
server_key_iv: "53394b4477a8394d0c00857d5ec18912"
server_key_version: "kuduclusterkey@0"
I20260925 16:40:44.373449 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:44.400390 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:44.404155 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:44.404352 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:44.405097 13466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:44.406234 13466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:44.406303 13466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:44.406363 13466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:44.406405 13466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:44.417827 13466 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37803
I20260925 16:40:44.419206 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:40:44.420411 13596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37803 every 8 connection(s)
I20260925 16:40:44.426344 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13466
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:44.438352 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:44.422539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:44787 (local address 127.0.97.254:38039)
0925 16:40:44.422675 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:44.422679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:44.424701 (+ 2022us) server_negotiation.cc:407] Connection header received
0925 16:40:44.426064 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:44.426069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:44.426132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:44.426202 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:44.427890 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.428763 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.430707 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.431092 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.431178 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:44.434138 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:44.434142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:44.434145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:44.434175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:44.435904 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.436563 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.436567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.436570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.436665 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.437060 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.437063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.437065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.437297 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:44.437392 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:44.438057 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 16:40:44.438185 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:40:44.440126 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:44.423071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38039 (local address 127.0.97.195:44787)
0925 16:40:44.424515 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:44.424574 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:44.425801 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:44.426650 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:44.426660 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:44.427019 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:44.427715 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:44.427735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.429118 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.429121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:44.430537 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:44.430548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.431389 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.431394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:44.431590 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:44.432286 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:44.432308 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:44.433937 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:44.436041 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:44.436047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:44.436063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:44.436436 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:44.436769 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:44.436772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:44.436774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:44.436941 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:44.437467 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:44.437473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:44.437774 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 16:40:44.438600 (+ 826us) client_negotiation.cc:240] Negotiation successful
0925 16:40:44.438956 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":131,"threads_started":1}
I20260925 16:40:44.441406 13597 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38039
I20260925 16:40:44.441675 13597 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.442320 13597 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:44.444064 13597 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:38039 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:40:44.444206 13597 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.444411 13597 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:44.444907 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:44.427004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.0.97.254:38039)
0925 16:40:44.429151 (+ 2147us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:44.429155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:44.429582 (+ 427us) server_negotiation.cc:407] Connection header received
0925 16:40:44.429624 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:44.429627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:44.429673 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:44.429751 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:44.433468 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.434392 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.439631 (+ 5239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:44.439867 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:44.439984 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:44.441367 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:44.441373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:44.441378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:44.441404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:44.443104 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.443616 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.443625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.443626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.443695 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:44.443984 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:44.443987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:44.443989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:44.444189 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:44.444400 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:44.444595 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 16:40:44.444712 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2068,"thread_start_us":89,"threads_started":1}
I20260925 16:40:44.445256 13597 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.445478 13597 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
W20260925 16:40:44.446326 13597 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 16:40:44 dist-test-slave-kxl8 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:44.935698 13329 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:44.935995 13329 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:44.937202 13131 ts_manager.cc:194] Registered new tserver with Master: 3aeccf5966eb47a1a83da5389cc68286 (127.0.97.193:34019)
I20260925 16:40:44.938961 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:36585
Sep 25 16:40:45 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:45.185073 13463 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:45.185398 13463 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:45.186223 13131 ts_manager.cc:194] Registered new tserver with Master: 5965656ba0514a2fa850b92511b25dc5 (127.0.97.194:44171)
I20260925 16:40:45.187031 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:37635
Sep 25 16:40:45 dist-test-slave-kxl8 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for HTTP/127.0.97.212@KRBTEST.COM
I20260925 16:40:45.446575 13597 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:45.446897 13597 heartbeater.cc:511] Master 127.0.97.254:38039 requested a full tablet report, sending...
I20260925 16:40:45.447714 13131 ts_manager.cc:194] Registered new tserver with Master: 6f9e365a20d64b6db5a0c8d8dddb9e82 (127.0.97.195:37803)
I20260925 16:40:45.448493 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:44787
I20260925 16:40:45.449824 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:40:45.469412 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.453771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.0.97.254:38039)
0925 16:40:45.458286 (+ 4515us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.458291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.458607 (+ 316us) server_negotiation.cc:407] Connection header received
0925 16:40:45.458660 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.458663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.458995 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.459461 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:45.460064 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.461714 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.462866 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.463232 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.463327 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.465410 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:45.465414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:45.465417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:45.465449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:45.467343 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:45.467844 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:45.467848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:45.467850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:45.467972 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:45.468296 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:45.468301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:45.468303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:45.468546 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:45.468654 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.469033 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.469174 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4169,"thread_start_us":90,"threads_started":1}
I20260925 16:40:45.475893 13130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:40:45.482025 13130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:40:45.529181 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.510425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34019 (local address 127.0.0.1:56630)
0925 16:40:45.522185 (+ 11760us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.522197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.522279 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.523995 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:45.523997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:45.524017 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:45.524102 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.524107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.525545 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.525548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.526937 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.526948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.528596 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.528600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.528718 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.528761 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 16:40:45.528880 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:40:45.529010 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11702,"thread_start_us":97,"threads_started":1}
I20260925 16:40:45.530988 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.510715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.0.97.193:34019)
0925 16:40:45.522551 (+ 11836us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.522558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.522581 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:40:45.522655 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.522660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.522866 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.523005 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:45.524284 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.525426 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.527406 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.528259 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.528368 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.528529 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.529875 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.530840 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11716,"thread_start_us":98,"threads_started":1}
I20260925 16:40:45.534472 13263 tablet_service.cc:1467] Processing CreateTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 (DEFAULT_TABLE table=test-table [id=84242adb08b248c39f9c35a456d87b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:45.535122 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.511522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.97.195:37803)
0925 16:40:45.521524 (+ 10002us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.521533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.527272 (+ 5739us) server_negotiation.cc:407] Connection header received
0925 16:40:45.527407 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.527413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.527630 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.527789 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:45.528204 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.529364 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.533379 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.534238 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.534327 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.534476 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.534867 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.534978 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9863,"thread_start_us":74,"threads_started":1}
I20260925 16:40:45.535561 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.511153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37803 (local address 127.0.0.1:44006)
0925 16:40:45.527163 (+ 16010us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.527177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.527277 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.527943 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:45.527946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:45.527965 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:45.528055 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.528061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.529512 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.529515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.530677 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.530688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.534585 (+ 3897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.534588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.534697 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.534722 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:40:45.535335 (+ 613us) client_negotiation.cc:240] Negotiation successful
0925 16:40:45.535418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15948,"thread_start_us":81,"threads_started":1}
I20260925 16:40:45.535652 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261aa3b5ca3c49b69630bc40d302dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:45.538430 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.508288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.97.194:44171)
0925 16:40:45.517235 (+ 8947us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.517243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.521878 (+ 4635us) server_negotiation.cc:407] Connection header received
0925 16:40:45.522096 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.522104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.522415 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.522579 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:45.524059 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.525329 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.536760 (+ 11431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.537601 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.537688 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.537830 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.538194 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.538293 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8807,"thread_start_us":133,"threads_started":1}
I20260925 16:40:45.538854 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.508139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:44171 (local address 127.0.0.1:37118)
0925 16:40:45.521693 (+ 13554us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.521768 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.521938 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.523723 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:45.523731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:45.523767 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:45.523920 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.523931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.535488 (+ 11557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.535491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.536579 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.536589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.537931 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.537935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.538032 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.538056 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:40:45.538624 (+ 568us) client_negotiation.cc:240] Negotiation successful
0925 16:40:45.538709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13315,"thread_start_us":101,"threads_started":1}
I20260925 16:40:45.538453 13531 tablet_service.cc:1467] Processing CreateTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 (DEFAULT_TABLE table=test-table [id=84242adb08b248c39f9c35a456d87b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:45.541520 13392 tablet_service.cc:1467] Processing CreateTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 (DEFAULT_TABLE table=test-table [id=84242adb08b248c39f9c35a456d87b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:45.542537 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261aa3b5ca3c49b69630bc40d302dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:45.539521 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261aa3b5ca3c49b69630bc40d302dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:45.573590 13633 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Bootstrap starting.
I20260925 16:40:45.576093 13633 tablet_bootstrap.cc:654] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:45.576531 13635 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Bootstrap starting.
I20260925 16:40:45.576912 13634 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Bootstrap starting.
I20260925 16:40:45.580076 13633 log.cc:824] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:45.582168 13634 tablet_bootstrap.cc:654] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:45.582731 13635 tablet_bootstrap.cc:654] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:45.583611 13634 log.cc:824] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:45.584445 13635 log.cc:824] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:45.588599 13633 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: No bootstrap required, opened a new log
I20260925 16:40:45.588860 13633 ts_tablet_manager.cc:1397] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260925 16:40:45.595749 13635 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: No bootstrap required, opened a new log
I20260925 16:40:45.596061 13635 ts_tablet_manager.cc:1397] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260925 16:40:45.592926 13633 raft_consensus.cc:348] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.597352 13633 raft_consensus.cc:374] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:45.597415 13633 raft_consensus.cc:729] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9e365a20d64b6db5a0c8d8dddb9e82, State: Initialized, Role: FOLLOWER
I20260925 16:40:45.597944 13633 consensus_queue.cc:260] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.599056 13633 ts_tablet_manager.cc:1428] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260925 16:40:45.599509 13597 heartbeater.cc:503] Master 127.0.97.254:38039 was elected leader, sending a full tablet report...
I20260925 16:40:45.602514 13635 raft_consensus.cc:348] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.602797 13635 raft_consensus.cc:374] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:45.602850 13635 raft_consensus.cc:729] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5965656ba0514a2fa850b92511b25dc5, State: Initialized, Role: FOLLOWER
I20260925 16:40:45.603339 13634 tablet_bootstrap.cc:492] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: No bootstrap required, opened a new log
I20260925 16:40:45.603328 13635 consensus_queue.cc:260] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.603657 13634 ts_tablet_manager.cc:1397] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20260925 16:40:45.604400 13635 ts_tablet_manager.cc:1428] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:40:45.604727 13463 heartbeater.cc:503] Master 127.0.97.254:38039 was elected leader, sending a full tablet report...
I20260925 16:40:45.610357 13634 raft_consensus.cc:348] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.610700 13634 raft_consensus.cc:374] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:45.610802 13634 raft_consensus.cc:729] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aeccf5966eb47a1a83da5389cc68286, State: Initialized, Role: FOLLOWER
I20260925 16:40:45.611411 13634 consensus_queue.cc:260] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.612358 13329 heartbeater.cc:503] Master 127.0.97.254:38039 was elected leader, sending a full tablet report...
I20260925 16:40:45.612591 13634 ts_tablet_manager.cc:1428] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260925 16:40:45.624199 13639 raft_consensus.cc:482] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:45.624452 13639 raft_consensus.cc:504] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.625811 13639 leader_election.cc:288] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5965656ba0514a2fa850b92511b25dc5 (127.0.97.194:44171), 3aeccf5966eb47a1a83da5389cc68286 (127.0.97.193:34019)
I20260925 16:40:45.641579 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.626994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:39227 (local address 127.0.97.193:34019)
0925 16:40:45.627108 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.627112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.630393 (+ 3281us) server_negotiation.cc:407] Connection header received
0925 16:40:45.636883 (+ 6490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.636888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.636947 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.636998 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:45.637769 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.638548 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.640177 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.640801 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.640931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.640986 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.641362 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.641443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:40:45.642308 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.626590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34019 (local address 127.0.97.195:39227)
0925 16:40:45.630315 (+ 3725us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.630327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.630472 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.637506 (+ 7034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:45.637509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:45.637528 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:45.637639 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.637646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.638695 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.638698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.640028 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.640039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.641050 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.641054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.641163 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.641201 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:40:45.642058 (+ 857us) client_negotiation.cc:240] Negotiation successful
0925 16:40:45.642136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3657,"thread_start_us":98,"threads_started":1}
I20260925 16:40:45.643121 13283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261aa3b5ca3c49b69630bc40d302dd17" candidate_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aeccf5966eb47a1a83da5389cc68286" is_pre_election: true
I20260925 16:40:45.643471 13283 raft_consensus.cc:2454] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9e365a20d64b6db5a0c8d8dddb9e82 in term 0.
I20260925 16:40:45.643772 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.627217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:55429 (local address 127.0.97.194:44171)
0925 16:40:45.627342 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.627346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.630160 (+ 2814us) server_negotiation.cc:407] Connection header received
0925 16:40:45.630689 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.630693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.630749 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.630802 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:45.640206 (+ 9404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.640955 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.642502 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.643089 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.643182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.643229 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.643580 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.643657 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:40:45.644172 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.626163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:44171 (local address 127.0.97.195:55429)
0925 16:40:45.630067 (+ 3904us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.630083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.630569 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.637146 (+ 6577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:45.637150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:45.637170 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:45.637354 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.637362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.641116 (+ 3754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.641119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.642352 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:45.642364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.643337 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.643340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:45.643437 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.643459 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:40:45.643966 (+ 507us) client_negotiation.cc:240] Negotiation successful
0925 16:40:45.644037 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3831,"thread_start_us":133,"threads_started":1}
I20260925 16:40:45.644219 13484 leader_election.cc:302] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aeccf5966eb47a1a83da5389cc68286, 6f9e365a20d64b6db5a0c8d8dddb9e82; no voters:
I20260925 16:40:45.644526 13639 raft_consensus.cc:2781] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:40:45.644598 13639 raft_consensus.cc:482] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:40:45.644637 13639 raft_consensus.cc:3037] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:45.644717 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261aa3b5ca3c49b69630bc40d302dd17" candidate_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5965656ba0514a2fa850b92511b25dc5" is_pre_election: true
I20260925 16:40:45.644964 13416 raft_consensus.cc:2454] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9e365a20d64b6db5a0c8d8dddb9e82 in term 0.
I20260925 16:40:45.645965 13639 raft_consensus.cc:504] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.646328 13639 leader_election.cc:288] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [CANDIDATE]: Term 1 election: Requested vote from peers 5965656ba0514a2fa850b92511b25dc5 (127.0.97.194:44171), 3aeccf5966eb47a1a83da5389cc68286 (127.0.97.193:34019)
W20260925 16:40:45.646986 13330 tablet.cc:2369] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:40:45.647231 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261aa3b5ca3c49b69630bc40d302dd17" candidate_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5965656ba0514a2fa850b92511b25dc5"
I20260925 16:40:45.647346 13416 raft_consensus.cc:3037] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:45.648626 13416 raft_consensus.cc:2454] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9e365a20d64b6db5a0c8d8dddb9e82 in term 1.
I20260925 16:40:45.648782 13283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261aa3b5ca3c49b69630bc40d302dd17" candidate_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aeccf5966eb47a1a83da5389cc68286"
I20260925 16:40:45.648900 13283 raft_consensus.cc:3037] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:45.649021 13486 leader_election.cc:302] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5965656ba0514a2fa850b92511b25dc5, 6f9e365a20d64b6db5a0c8d8dddb9e82; no voters:
I20260925 16:40:45.649250 13639 raft_consensus.cc:2781] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:45.649489 13639 raft_consensus.cc:686] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9e365a20d64b6db5a0c8d8dddb9e82, State: Running, Role: LEADER
I20260925 16:40:45.649854 13639 consensus_queue.cc:237] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } }
I20260925 16:40:45.650287 13283 raft_consensus.cc:2454] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9e365a20d64b6db5a0c8d8dddb9e82 in term 1.
I20260925 16:40:45.653815 13131 catalog_manager.cc:5495] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9e365a20d64b6db5a0c8d8dddb9e82 (127.0.97.195). New cstate: current_term: 1 leader_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9e365a20d64b6db5a0c8d8dddb9e82" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 } health_report { overall_health: UNKNOWN } } }
W20260925 16:40:45.661151 13464 tablet.cc:2369] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:40:45.667910 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.663289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.0.97.195:37803)
0925 16:40:45.663406 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:45.663411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:45.663427 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:40:45.663513 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:45.663517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:45.663586 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:45.663706 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:40:45.663988 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.664758 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.666050 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:45.666417 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:45.666525 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:45.666883 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:40:45.667448 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:40:45.667575 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:45.667670 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 16:40:45.667733 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 16:40:45.672583 13598 tablet.cc:2369] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:40:45.675350 13283 raft_consensus.cc:1261] T 261aa3b5ca3c49b69630bc40d302dd17 P 3aeccf5966eb47a1a83da5389cc68286 [term 1 FOLLOWER]: Refusing update from remote peer 6f9e365a20d64b6db5a0c8d8dddb9e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:45.675690 13416 raft_consensus.cc:1261] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 1 FOLLOWER]: Refusing update from remote peer 6f9e365a20d64b6db5a0c8d8dddb9e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:45.676079 13644 consensus_queue.cc:1035] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aeccf5966eb47a1a83da5389cc68286" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:45.676445 13639 consensus_queue.cc:1035] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5965656ba0514a2fa850b92511b25dc5" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:45.689790 13647 mvcc.cc:205] Tried to move back new op lower bound from 7333291809477918720 to 7333291809386393600. Current Snapshot: MvccSnapshot[applied={T|T < 7333291809477918720}]
I20260925 16:40:45.689967 13649 mvcc.cc:205] Tried to move back new op lower bound from 7333291809477918720 to 7333291809386393600. Current Snapshot: MvccSnapshot[applied={T|T < 7333291809477918720}]
I20260925 16:40:45.690371 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333291809477918720 to 7333291809386393600. Current Snapshot: MvccSnapshot[applied={T|T < 7333291809477918720}]
I20260925 16:40:45.698928 13131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:45.699111 13131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:45.702004 13131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83d386b9114d4fa981c7cd1daab344e0: Sending DeleteTablet for 3 replicas of tablet 261aa3b5ca3c49b69630bc40d302dd17
I20260925 16:40:45.702935 13263 tablet_service.cc:1514] Processing DeleteTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:45 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:56630
I20260925 16:40:45.703022 13392 tablet_service.cc:1514] Processing DeleteTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:45 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:37118
I20260925 16:40:45.704048 13531 tablet_service.cc:1514] Processing DeleteTablet for tablet 261aa3b5ca3c49b69630bc40d302dd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:45 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:44006
I20260925 16:40:45.705139 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13169
I20260925 16:40:45.709368 13661 tablet_replica.cc:330] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: stopping tablet replica
I20260925 16:40:45.709753 13661 raft_consensus.cc:2229] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:40:45.710084 13659 tablet_replica.cc:330] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: stopping tablet replica
I20260925 16:40:45.710387 13659 raft_consensus.cc:2229] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:40:45.710729 13659 raft_consensus.cc:2258] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:45.711305 13661 raft_consensus.cc:2258] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:45.713018 13661 ts_tablet_manager.cc:1905] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:45.715996 13661 ts_tablet_manager.cc:1918] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:45.716100 13661 log.cc:1196] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/261aa3b5ca3c49b69630bc40d302dd17
I20260925 16:40:45.716447 13661 ts_tablet_manager.cc:1939] T 261aa3b5ca3c49b69630bc40d302dd17 P 6f9e365a20d64b6db5a0c8d8dddb9e82: Deleting consensus metadata
I20260925 16:40:45.717348 13659 ts_tablet_manager.cc:1905] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:45.717767 13117 catalog_manager.cc:4841] TS 6f9e365a20d64b6db5a0c8d8dddb9e82 (127.0.97.195:37803): tablet 261aa3b5ca3c49b69630bc40d302dd17 (table test-table [id=84242adb08b248c39f9c35a456d87b04]) successfully deleted
I20260925 16:40:45.720675 13659 ts_tablet_manager.cc:1918] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:45.720778 13659 log.cc:1196] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/261aa3b5ca3c49b69630bc40d302dd17
I20260925 16:40:45.721107 13659 ts_tablet_manager.cc:1939] T 261aa3b5ca3c49b69630bc40d302dd17 P 5965656ba0514a2fa850b92511b25dc5: Deleting consensus metadata
W20260925 16:40:45.723048 13115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:34019 (error 108)
I20260925 16:40:45.724063 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13331
I20260925 16:40:45.731544 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:45.731224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:34019 (local address 127.0.0.1:56632)
0925 16:40:45.731371 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.731451 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 16:40:45.731777 13115 catalog_manager.cc:4584] TS 3aeccf5966eb47a1a83da5389cc68286 (127.0.97.193:34019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 261aa3b5ca3c49b69630bc40d302dd17: Network error: Client connection negotiation failed: client connection to 127.0.97.193:34019: connect: Connection refused (error 111)
W20260925 16:40:45.734692 13629 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:40:45.734060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:44171 (local address 127.0.0.1:37120)
0925 16:40:45.734213 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:45.734227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:45.734313 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:45.734598 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:44171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 16:40:45.735317 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13466
I20260925 16:40:45.740933 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13098
I20260925 16:40:45.747087 391 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:40:46.081966 391 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:40:46.082145 391 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:40:46.427155 391 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:40:46.427 UTC [12165] LOG: received smart shutdown request
2026-09-25 16:40:46.431 UTC [12165] LOG: background worker "logical replication launcher" (PID 12174) exited with exit code 1
2026-09-25 16:40:46.431 UTC [12169] LOG: shutting down
2026-09-25 16:40:46.624 UTC [12165] LOG: database system is shut down
2026-09-25T16:40:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48973 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:40:46 dist-test-slave-kxl8 krb5kdc[13676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:40:46 dist-test-slave-kxl8 krb5kdc[13676](info): set up 2 sockets
Sep 25 16:40:46 dist-test-slave-kxl8 krb5kdc[13676](info): commencing operation
krb5kdc: starting...
W20260925 16:40:47.837805 391 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:40:47 dist-test-slave-kxl8 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:40:47.958685 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42635
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:48.069430 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:48.069686 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:48.069725 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:48.073625 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:48.073674 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:48.073693 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:48.073710 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:48.073725 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:48.078748 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:48.079695 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:48.080631 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:48.088881 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:48.088876 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:48.088958 13689 server_base.cc:1034] running on GCE node
W20260925 16:40:48.089156 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:48.089700 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:48.090670 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:48.091827 13689 hybrid_clock.cc:648] HybridClock initialized: now 1790354448091809 us; error 31 us; skew 500 ppm
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:48.094782 13689 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:48.095712 13689 webserver.cc:460] Webserver started at http://127.0.97.254:35815/ using document root <none> and password file <none>
I20260925 16:40:48.096339 13689 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:48.096421 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:48.096596 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:48.098696 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d8bb787906794d6c94250a1fad149f85"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "8f99bec2dd671fccf456f7322af40f8b"
server_key_iv: "19ee41f3e4d8b70c2fade3c8823f3eea"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.099207 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d8bb787906794d6c94250a1fad149f85"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "8f99bec2dd671fccf456f7322af40f8b"
server_key_iv: "19ee41f3e4d8b70c2fade3c8823f3eea"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.103116 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:40:48.105531 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:48.106755 13689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:40:48.106899 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d8bb787906794d6c94250a1fad149f85"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "8f99bec2dd671fccf456f7322af40f8b"
server_key_iv: "19ee41f3e4d8b70c2fade3c8823f3eea"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.107009 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:48.123963 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:48.127321 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:48.127525 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:48.134457 13689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42635
I20260925 16:40:48.134501 13756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42635 every 8 connection(s)
I20260925 16:40:48.135406 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:48.138132 13757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:48.141827 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.144794 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13689
I20260925 16:40:48.144936 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:40:48.145200 391 external_mini_cluster.cc:1428] Setting key a5b394e8f74d35e6de7cdd1800de25a1
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.253@KRBTEST.COM" created.
I20260925 16:40:48.157404 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.143943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.97.254:42635)
0925 16:40:48.144358 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.144372 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.144606 (+ 234us) server_negotiation.cc:407] Connection header received
0925 16:40:48.145744 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.145753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.145990 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.146191 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.147081 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.147914 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.149501 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.149857 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.149966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.152621 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.152626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.152635 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.152653 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.154296 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.154811 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.154816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.154818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.154921 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.155229 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.155232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.155234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.155575 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.155707 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.156167 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.156354 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260925 16:40:48.157401 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.143943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42635 (local address 127.0.0.1:57800)
0925 16:40:48.144407 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.144454 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.145430 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.146223 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.146227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.146558 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.146909 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.146932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.148042 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.148046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.149362 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.149376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.150019 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.150025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.150191 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.150795 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.150822 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.152478 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.154421 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.154428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.154437 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.154701 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.155014 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.155019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.155021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.155135 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.155693 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.155698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.155837 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.156031 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.156438 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":97,"thread_start_us":145,"threads_started":1}
I20260925 16:40:48.160248 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.161173 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.160768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57852)
0925 16:40:48.160934 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.161057 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 16:40:48.161422 13706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
I20260925 16:40:48.162323 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.162039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57860)
0925 16:40:48.162190 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.162246 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 16:40:48.162545 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40561: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
I20260925 16:40:48.172678 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40561
--webserver_interface=127.0.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:48.211799 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } attempt: 1
I20260925 16:40:48.212741 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.212441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57876)
0925 16:40:48.212599 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.212665 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 16:40:48.213678 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.213473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57892)
0925 16:40:48.213574 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.213619 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:40:48.213869 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40561: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
I20260925 16:40:48.282058 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } attempt: 2
I20260925 16:40:48.283020 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.282699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57904)
0925 16:40:48.282877 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.282942 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 16:40:48.283808 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.283562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57920)
0925 16:40:48.283702 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.283749 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 16:40:48.283962 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40561: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40561: connect: Connection refused (error 111)
W20260925 16:40:48.285794 13764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:48.286029 13764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:48.286082 13764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:48.290220 13764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:48.290272 13764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:48.290292 13764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:48.290309 13764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:48.290356 13764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:48.295692 13764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:48.296736 13764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:48.297793 13764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:48.306123 13770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:48.306200 13772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:48.306334 13769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:48.306488 13764 server_base.cc:1034] running on GCE node
I20260925 16:40:48.307058 13764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:48.308142 13764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:48.309365 13764 hybrid_clock.cc:648] HybridClock initialized: now 1790354448309326 us; error 51 us; skew 500 ppm
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:48.312147 13764 init.cc:377] Logged in from keytab as kudu/127.0.97.253@KRBTEST.COM (short username kudu)
I20260925 16:40:48.312963 13764 webserver.cc:460] Webserver started at http://127.0.97.253:34841/ using document root <none> and password file <none>
I20260925 16:40:48.313747 13764 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:48.313858 13764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:48.314090 13764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:48.316095 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data/instance:
uuid: "3881f773a71948fbaa07f0893d1b4e67"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "fb308aca8a09b6414fd7fd276b0389c6"
server_key_iv: "1f15433dd2ae395a6ce6133e8a5e4a83"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.316655 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal/instance:
uuid: "3881f773a71948fbaa07f0893d1b4e67"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "fb308aca8a09b6414fd7fd276b0389c6"
server_key_iv: "1f15433dd2ae395a6ce6133e8a5e4a83"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.320149 13764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:40:48.322485 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:48.323663 13764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:40:48.323802 13764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "3881f773a71948fbaa07f0893d1b4e67"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "fb308aca8a09b6414fd7fd276b0389c6"
server_key_iv: "1f15433dd2ae395a6ce6133e8a5e4a83"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.323908 13764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:48.339001 13764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:48.341900 13764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:48.342105 13764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:48.349061 13764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.253:40561
I20260925 16:40:48.349076 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.253:40561 every 8 connection(s)
I20260925 16:40:48.350028 13764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
I20260925 16:40:48.352662 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:48.356184 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.358743 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13764
I20260925 16:40:48.358884 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-1/wal/instance
I20260925 16:40:48.359159 391 external_mini_cluster.cc:1428] Setting key d11aa0e0a0239c6b65fdd70d4129a3ec
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.252@KRBTEST.COM" created.
I20260925 16:40:48.369693 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.357958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.97.254:42635)
0925 16:40:48.358095 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.358099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.358963 (+ 864us) server_negotiation.cc:407] Connection header received
0925 16:40:48.359979 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.359983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.360052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.360153 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.361304 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.361909 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.363102 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.363347 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.363461 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.366017 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.366020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.366022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.366052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.367564 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.368060 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.368064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.368066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.368155 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.368579 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.368582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.368583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.368765 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.368878 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.369472 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.369575 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:40:48.370834 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.358363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42635 (local address 127.0.0.1:57812)
0925 16:40:48.358812 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.358864 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.359759 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.360323 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.360330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.360667 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.361126 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.361138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.362050 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.362053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.362976 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.362986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.363496 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.363500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.363676 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.364324 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.364339 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.365845 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.367679 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.367685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.367698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.367950 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.368249 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.368253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.368255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.368455 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.368891 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.368899 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.369174 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.369403 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.369766 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260925 16:40:48.372283 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.375138 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } attempt: 3
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:40:48.385361 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.375536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57934)
0925 16:40:48.375689 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.375702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.375781 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.376202 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.376208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.376385 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.377027 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.377038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.377917 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.377922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.378560 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.378569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.378990 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.378996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.379176 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.379555 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.379567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.381093 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.383483 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.383488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.383490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.383787 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.384073 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.384076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.384078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.384170 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.384816 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.384820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.384937 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.385026 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.385161 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 16:40:48.385483 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.375614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.0.97.253:40561)
0925 16:40:48.375986 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.375994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.376005 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:40:48.376033 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.376035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.376068 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.376151 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.377163 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.377801 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.378658 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.378872 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.378947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.381325 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.381332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.381338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.381366 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.383373 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.383886 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.383891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.383893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.383974 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.384268 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.384271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.384272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.384658 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.384795 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.385125 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.385232 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":313,"spinlock_wait_cycles":4224,"thread_start_us":115,"threads_started":1}
I20260925 16:40:48.386034 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.372805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57926)
0925 16:40:48.372958 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.372974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.373049 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.373680 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.373682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.373814 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.373894 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.373899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.375422 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.375428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.376229 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.376236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.376812 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.376815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.376902 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.377435 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.377452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.380549 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.383500 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.383506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.383508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.383723 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.384026 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.384030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.384032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.384137 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.385555 (+ 1418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.385561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.385663 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.385767 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.385925 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 16:40:48.386174 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.372896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.97.253:40561)
0925 16:40:48.373213 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.373224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.373248 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:40:48.373327 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.373333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.373551 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.373702 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.374010 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.375316 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.376345 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.376689 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.376808 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.380694 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.380702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.380712 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.380732 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.383399 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.383829 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.383842 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.383843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.383933 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.384253 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.384257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.384260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.384658 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.384795 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.385833 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.385988 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":208,"spinlock_wait_cycles":22656,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
I20260925 16:40:48.387378 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.387477 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.388074 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.387839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49354)
0925 16:40:48.387955 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.388004 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 16:40:48.388136 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:46403
--webserver_interface=127.0.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:48.388387 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.387857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49358)
0925 16:40:48.388248 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.388326 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
W20260925 16:40:48.388620 13780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.388765 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.388521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49370)
0925 16:40:48.388665 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.388710 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 16:40:48.388906 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.389492 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.389236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49384)
0925 16:40:48.389369 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.389424 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 16:40:48.389784 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.420137 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 1
I20260925 16:40:48.421025 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.420728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49392)
0925 16:40:48.420885 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.420949 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 16:40:48.421777 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.421552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49400)
0925 16:40:48.421665 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.421717 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:40:48.421943 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.439069 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 1
I20260925 16:40:48.439981 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.439649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49416)
0925 16:40:48.439818 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.439888 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 16:40:48.440696 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.440487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49418)
0925 16:40:48.440588 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.440637 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:40:48.440832 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.497120 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 2
I20260925 16:40:48.498000 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.497712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49420)
0925 16:40:48.497863 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.497924 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 16:40:48.498663 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.498485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49432)
0925 16:40:48.498575 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.498616 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 16:40:48.498807 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
W20260925 16:40:48.499228 13840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:48.499452 13840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:48.499491 13840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:48.504877 13840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:48.504966 13840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:48.505003 13840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:48.505031 13840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:48.505081 13840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:48.508983 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 2
I20260925 16:40:48.510049 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.509493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49448)
0925 16:40:48.509888 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.509958 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260925 16:40:48.510872 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.510534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49464)
0925 16:40:48.510688 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.510773 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 16:40:48.511639 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:46403: Network error: Client connection negotiation failed: client connection to 127.0.97.252:46403: connect: Connection refused (error 111)
I20260925 16:40:48.512935 13840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42635,127.0.97.253:40561,127.0.97.252:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:48.514115 13840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:48.515123 13840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:48.524235 13845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:48.524386 13840 server_base.cc:1034] running on GCE node
W20260925 16:40:48.524564 13848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:48.524698 13846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:48.525233 13840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:48.526432 13840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:48.527588 13840 hybrid_clock.cc:648] HybridClock initialized: now 1790354448527569 us; error 34 us; skew 500 ppm
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:48.530429 13840 init.cc:377] Logged in from keytab as kudu/127.0.97.252@KRBTEST.COM (short username kudu)
I20260925 16:40:48.531229 13840 webserver.cc:460] Webserver started at http://127.0.97.252:44357/ using document root <none> and password file <none>
I20260925 16:40:48.531719 13840 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:48.531781 13840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:48.531937 13840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:48.533989 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data/instance:
uuid: "e9309c27218b47b48508725da389edd6"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "3d2312b3a2878a5601f99eb4df5ee62d"
server_key_iv: "b5f94b89daed03643ced0f7bb2ebf812"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.534487 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal/instance:
uuid: "e9309c27218b47b48508725da389edd6"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "3d2312b3a2878a5601f99eb4df5ee62d"
server_key_iv: "b5f94b89daed03643ced0f7bb2ebf812"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.538087 13840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:40:48.540351 13855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:48.541633 13840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:40:48.541765 13840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "e9309c27218b47b48508725da389edd6"
format_stamp: "Formatted at 2026-09-25 16:40:48 on dist-test-slave-kxl8"
server_key: "3d2312b3a2878a5601f99eb4df5ee62d"
server_key_iv: "b5f94b89daed03643ced0f7bb2ebf812"
server_key_version: "encryptionkey@0"
I20260925 16:40:48.541867 13840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:48.563915 13840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:48.567376 13840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:48.567579 13840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:48.574682 13840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.252:46403
I20260925 16:40:48.574777 13907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.252:46403 every 8 connection(s)
I20260925 16:40:48.575654 13840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
I20260925 16:40:48.578405 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:48.582000 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.584491 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13840
I20260925 16:40:48.584620 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790354288967115-391-0/minicluster-data/master-2/wal/instance
I20260925 16:40:48.584951 391 external_mini_cluster.cc:1428] Setting key 1709389988ada07c2bd3b49ef574cc07
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:40:48.596709 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.583788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.97.254:42635)
0925 16:40:48.583918 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.583922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.584790 (+ 868us) server_negotiation.cc:407] Connection header received
0925 16:40:48.585999 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.586003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.586082 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.586183 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.587406 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.588206 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.589529 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.589777 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.589880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.592787 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.592791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.592793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.592818 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.594474 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.595188 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.595192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.595194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.595293 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.595641 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.595644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.595646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.595836 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.595940 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.596408 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.596524 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:40:48.597595 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.584149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42635 (local address 127.0.0.1:57814)
0925 16:40:48.584626 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.584678 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.585760 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.586348 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.586355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.586779 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.587246 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.587267 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.588331 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.588334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.589421 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.589432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.589959 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.589962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.590174 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.590812 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.590837 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.592555 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.594600 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.594607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.594614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.594997 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.595388 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.595391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.595393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.595534 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.595949 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.595958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.596178 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.596352 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.596653 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":29,"spinlock_wait_cycles":3840,"thread_start_us":147,"threads_started":1}
I20260925 16:40:48.597605 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.585618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.97.252:46403)
0925 16:40:48.585899 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.585908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.586095 (+ 187us) server_negotiation.cc:407] Connection header received
0925 16:40:48.586335 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.586339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.586500 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.586634 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.587511 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.588618 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.589764 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.590191 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.590274 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.593065 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.593071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.593080 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.593099 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.594992 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.595408 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.595418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.595420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.595520 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.595819 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.595822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.595824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.596158 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.596367 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.596570 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.596705 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260925 16:40:48.597951 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 3
I20260925 16:40:48.599486 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.602873 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } attempt: 3
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:40:48.608085 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.598359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49480)
0925 16:40:48.598476 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.598490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.598569 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.598823 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.598826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.598988 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.599106 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.599114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.599995 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.600001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.600854 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.600868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.601394 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.601398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.601521 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.602026 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.602040 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.603498 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.606094 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.606099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.606101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.606312 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.606589 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.606594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.606596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.606687 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.607154 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.607158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.607253 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.607329 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.607448 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":21}
I20260925 16:40:48.609627 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.598423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.97.252:46403)
0925 16:40:48.598533 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.598537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.598550 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:40:48.598671 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.598674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.598710 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.599211 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.599281 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.599856 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.600978 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.601261 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.601385 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.603656 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.603659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.603662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.603683 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.605986 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.606425 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.606428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.606429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.606495 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.606792 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.606795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.606797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.607043 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.609187 (+ 2144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.609291 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.609427 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:40:48.611047 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.600307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.97.254:42635)
0925 16:40:48.600422 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.600424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.600448 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:40:48.600472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.600474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.600503 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.600556 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.601256 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.602147 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.603267 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.603883 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.603952 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.607361 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.607365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.607367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.607392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.609080 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.609592 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.609595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.609597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.609674 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.609995 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.610001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.610005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.610213 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.610406 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.610770 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.610887 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260925 16:40:48.611245 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.601101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.97.253:40561)
0925 16:40:48.601258 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.601262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.601298 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:40:48.601342 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.601347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.601397 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.601472 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.602024 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.602527 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.603576 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.604033 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.604116 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.607324 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.607329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.607331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.607356 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.609397 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.609892 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.609896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.609897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.609978 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.610295 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.610298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.610300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.610510 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.610576 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.610942 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.611077 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:40:48.611683 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.599980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40561 (local address 127.0.0.1:57946)
0925 16:40:48.600062 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.600077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.600142 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.601583 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.601585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.601761 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.601875 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.601884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.602635 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.602641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.603453 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.603464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.604275 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.604277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.604357 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.605186 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.605197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.607164 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.609517 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.609521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.609523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.609785 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.610102 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.610104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.610106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.610183 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.610663 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.610670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.610766 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.611410 (+ 644us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.611540 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 16:40:48.613531 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: Bootstrap starting.
I20260925 16:40:48.614181 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:40:48.616418 13757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:48.616930 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.603215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49486)
0925 16:40:48.603440 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.603454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.603528 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.604791 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.604796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.604985 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.605059 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.605064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.606698 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.606702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.607471 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.607482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.608146 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.608153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.608247 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.608791 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.608810 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.610304 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.615169 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.615172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.615174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.615413 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.615715 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.615718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.615720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.615804 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.616463 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.616472 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.616568 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.616663 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.616788 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 16:40:48.617367 13757 log.cc:824] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:48.617838 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.603431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.0.97.252:46403)
0925 16:40:48.604579 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.604587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.604599 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:40:48.604632 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.604635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.604674 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.604766 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.605154 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.606527 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.607600 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.608015 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.608440 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.610460 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.610463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.610465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.610487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.615064 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.615525 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.615528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.615530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.615614 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.615913 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.615916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.615917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.616098 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.617426 (+ 1328us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.617546 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.617654 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":124,"threads_started":1}
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:40:48.619505 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: No bootstrap required, opened a new log
Sep 25 16:40:48 dist-test-slave-kxl8 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:40:48.621109 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: Bootstrap starting.
I20260925 16:40:48.623163 13757 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.623385 13757 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:48.623453 13757 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bb787906794d6c94250a1fad149f85, State: Initialized, Role: FOLLOWER
I20260925 16:40:48.623819 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:48.623963 13757 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.624468 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.624648 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.624670 13832 log.cc:824] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:48.624853 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.614580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:46403 (local address 127.0.0.1:49490)
0925 16:40:48.614698 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:48.614712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:48.614785 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:48.615436 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:48.615442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:48.615593 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:48.615665 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.615671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.616578 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.616582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.617523 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:48.617535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.618193 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.618196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:48.618302 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.618926 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:48.618939 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:48.621410 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:48.623325 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.623329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.623331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.623573 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.623879 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:48.623882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:48.623884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:48.623974 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:48.624421 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:48.624423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:48.624501 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 16:40:48.624605 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:40:48.624719 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 16:40:48.625372 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.614766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.0.97.252:46403)
0925 16:40:48.615211 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.615214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.615223 (+ 9us) server_negotiation.cc:407] Connection header received
0925 16:40:48.615248 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.615251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.615284 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.615343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.615769 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.616462 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.617649 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.617982 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.618084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.621540 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.621543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.621546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.621569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.623210 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.623676 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.623682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.623687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.623773 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.624089 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.624096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.624098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.624298 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.624388 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.625102 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.625204 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260925 16:40:48.625379 13757 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:48.627125 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.612532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.0.97.253:40561)
0925 16:40:48.612634 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.612638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.612714 (+ 76us) server_negotiation.cc:407] Connection header received
0925 16:40:48.612822 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.612826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.612861 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.612952 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.613657 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.614398 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.615465 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.615801 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.615866 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.618618 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.618622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.618624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.618646 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.620204 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.625489 (+ 5285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.625493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.625495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.625589 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.625917 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.625922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.625925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.626146 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.626311 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.626882 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.626986 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:40:48.627568 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: No bootstrap required, opened a new log
W20260925 16:40:48.628985 13710 tablet.cc:2369] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:40:48.629881 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: Bootstrap starting.
I20260925 16:40:48.632525 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 16:40:48.632656 13927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:40:48.632719 13927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:40:48.633142 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.633368 13832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:48.633420 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3881f773a71948fbaa07f0893d1b4e67, State: Initialized, Role: FOLLOWER
I20260925 16:40:48.633828 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.634255 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.634370 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.635134 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:48.634351 13689 master_runner.cc:186] Error getting master registration for 127.0.97.253:40561: OK, instance_id { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" instance_seqno: 1790354448347180 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:40:48.637228 13689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:40:48.638975 13908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:48.639876 13908 log.cc:824] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: Log is configured to *not* fsync() on all Append() calls
W20260925 16:40:48.639983 13945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:40:48.640107 13945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:40:48.643060 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: No bootstrap required, opened a new log
I20260925 16:40:48.641932 13764 master_runner.cc:186] Error getting master registration for 127.0.97.252:46403: OK, instance_id { permanent_uuid: "e9309c27218b47b48508725da389edd6" instance_seqno: 1790354448572549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:40:48.645210 13764 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:40:48.647866 13908 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.648289 13908 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:48.648393 13908 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9309c27218b47b48508725da389edd6, State: Initialized, Role: FOLLOWER
I20260925 16:40:48.649077 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.637402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.0.97.252:46403)
0925 16:40:48.637522 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.637527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.637541 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:40:48.637579 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.637583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.637632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.637712 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.641138 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.641872 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.642935 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.643231 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.643305 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.645185 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.645189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.645191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.645216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.647205 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.647690 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.647694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.647696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.647790 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.648107 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.648112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.648113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.648325 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.648432 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.648792 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.648899 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:40:48.648938 13908 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.650902 13908 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:48.658946 13840 master_runner.cc:263] No leader master found from master e9309c27218b47b48508725da389edd6
I20260925 16:40:48.659018 13840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:40:48.659111 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.641193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.0.97.253:40561)
0925 16:40:48.641316 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.641320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.649784 (+ 8464us) server_negotiation.cc:407] Connection header received
0925 16:40:48.649976 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.649979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.650037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.650105 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.651202 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.651944 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.653031 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.653406 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.653511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.655342 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.655346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.655349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.655376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.657109 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.657618 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.657621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.657623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.657709 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.658027 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.658030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.658032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.658279 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.658379 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.658775 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.658900 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:40:48.661967 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.662223 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.663527 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:48.640668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.0.97.254:42635)
0925 16:40:48.640783 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:48.640787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:48.649382 (+ 8595us) server_negotiation.cc:407] Connection header received
0925 16:40:48.649558 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:48.649561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:48.649619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:48.649671 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:48.650532 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.651301 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.652408 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:48.652789 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:48.652875 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:48.659672 (+ 6797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:48.659676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:48.659679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:48.659711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:48.661344 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.661829 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.661833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.661835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.661923 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:48.662232 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:48.662238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:48.662241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:48.662485 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:48.662619 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:48.663264 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 16:40:48.663382 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:40:48.666555 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:40:48.666697 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:40:48.687983 13915 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:48.688163 13915 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.690303 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bb787906794d6c94250a1fad149f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3881f773a71948fbaa07f0893d1b4e67" is_pre_election: true
I20260925 16:40:48.690629 13805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8bb787906794d6c94250a1fad149f85 in term 0.
I20260925 16:40:48.691155 13706 leader_election.cc:302] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3881f773a71948fbaa07f0893d1b4e67, d8bb787906794d6c94250a1fad149f85; no voters:
I20260925 16:40:48.692008 13882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bb787906794d6c94250a1fad149f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9309c27218b47b48508725da389edd6" is_pre_election: true
I20260925 16:40:48.692306 13882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8bb787906794d6c94250a1fad149f85 in term 0.
I20260925 16:40:48.689208 13915 leader_election.cc:288] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3881f773a71948fbaa07f0893d1b4e67 (127.0.97.253:40561), e9309c27218b47b48508725da389edd6 (127.0.97.252:46403)
I20260925 16:40:48.694324 13915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:40:48.694401 13915 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:40:48.694442 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:48.695798 13915 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.696693 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bb787906794d6c94250a1fad149f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3881f773a71948fbaa07f0893d1b4e67"
I20260925 16:40:48.696825 13805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:48.697574 13882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bb787906794d6c94250a1fad149f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9309c27218b47b48508725da389edd6"
I20260925 16:40:48.697693 13882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:48.698956 13882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8bb787906794d6c94250a1fad149f85 in term 1.
I20260925 16:40:48.699380 13705 leader_election.cc:302] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8bb787906794d6c94250a1fad149f85, e9309c27218b47b48508725da389edd6; no voters:
I20260925 16:40:48.700489 13805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8bb787906794d6c94250a1fad149f85 in term 1.
I20260925 16:40:48.700906 13915 leader_election.cc:288] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [CANDIDATE]: Term 1 election: Requested vote from peers 3881f773a71948fbaa07f0893d1b4e67 (127.0.97.253:40561), e9309c27218b47b48508725da389edd6 (127.0.97.252:46403)
I20260925 16:40:48.701108 13915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:48.701226 13915 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [term 1 LEADER]: Becoming Leader. State: Replica: d8bb787906794d6c94250a1fad149f85, State: Running, Role: LEADER
I20260925 16:40:48.701669 13915 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } }
I20260925 16:40:48.707072 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bb787906794d6c94250a1fad149f85. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.707296 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:48.707808 13965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:48.708813 13965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:48.716545 13882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [term 1 FOLLOWER]: Refusing update from remote peer d8bb787906794d6c94250a1fad149f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:48.719182 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:48.723695 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bb787906794d6c94250a1fad149f85. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.723834 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.724956 13805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [term 1 FOLLOWER]: Refusing update from remote peer d8bb787906794d6c94250a1fad149f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:48.725598 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:48.729730 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bb787906794d6c94250a1fad149f85. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.729920 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.736639 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.736773 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:48.738011 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.738118 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.738539 13966 mvcc.cc:205] Tried to move back new op lower bound from 7333291821934981120 to 7333291821895557120. Current Snapshot: MvccSnapshot[applied={T|T < 7333291821934981120}]
I20260925 16:40:48.740248 13969 mvcc.cc:205] Tried to move back new op lower bound from 7333291821934981120 to 7333291821895557120. Current Snapshot: MvccSnapshot[applied={T|T < 7333291821934981120}]
I20260925 16:40:48.742641 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.742753 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:48.743880 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8bb787906794d6c94250a1fad149f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bb787906794d6c94250a1fad149f85" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42635 } } peers { permanent_uuid: "3881f773a71948fbaa07f0893d1b4e67" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40561 } } peers { permanent_uuid: "e9309c27218b47b48508725da389edd6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 46403 } } }
I20260925 16:40:48.744010 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:40:48.749809 13967 mvcc.cc:205] Tried to move back new op lower bound from 7333291821934981120 to 7333291821895557120. Current Snapshot: MvccSnapshot[applied={T|T < 7333291821934981120}]
I20260925 16:40:48.750342 13965 catalog_manager.cc:1347] Generated new cluster ID: 86cf6f905066445898409136823c67db
I20260925 16:40:48.750397 13965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:48.768841 13965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:40:48.770205 13965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:48.787371 13965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8bb787906794d6c94250a1fad149f85: Generated new TSK 0
I20260925 16:40:48.788305 13965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:40:49.641404 13945 catalog_manager.cc:1259] Loaded cluster ID: 86cf6f905066445898409136823c67db
I20260925 16:40:49.641477 13945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: loading cluster ID for follower catalog manager: success
I20260925 16:40:49.643160 13945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: acquiring CA information for follower catalog manager: success
I20260925 16:40:49.644301 13945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3881f773a71948fbaa07f0893d1b4e67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:40:49.668143 13961 catalog_manager.cc:1259] Loaded cluster ID: 86cf6f905066445898409136823c67db
I20260925 16:40:49.668242 13961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: loading cluster ID for follower catalog manager: success
I20260925 16:40:49.669993 13961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: acquiring CA information for follower catalog manager: success
I20260925 16:40:49.671192 13961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9309c27218b47b48508725da389edd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:40:49.850304 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13689
I20260925 16:40:49.856664 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13764
I20260925 16:40:49.862505 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13840
2026-09-25T16:40:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3121 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:40:49 dist-test-slave-kxl8 krb5kdc[13983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:40:49 dist-test-slave-kxl8 krb5kdc[13983](info): set up 2 sockets
Sep 25 16:40:49 dist-test-slave-kxl8 krb5kdc[13983](info): commencing operation
krb5kdc: starting...
W20260925 16:40:50.897505 391 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:40:50 dist-test-slave-kxl8 krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:40:51.017084 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42177
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:51.125725 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:51.126011 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:51.126085 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:51.130148 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:51.130228 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:51.130265 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:51.130323 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:51.130358 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:51.135664 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:51.136674 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:51.137702 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:51.146306 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:51.146314 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:51.146350 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:51.146311 13996 server_base.cc:1034] running on GCE node
I20260925 16:40:51.147102 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:51.148198 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:51.149372 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790354451149351 us; error 34 us; skew 500 ppm
Sep 25 16:40:51 dist-test-slave-kxl8 krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:51.152302 13996 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:51.153157 13996 webserver.cc:460] Webserver started at http://127.0.97.254:45641/ using document root <none> and password file <none>
I20260925 16:40:51.153753 13996 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:51.153841 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:51.154066 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:51.156075 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "43693605520f4d5ab45f3a54f5199cbf"
format_stamp: "Formatted at 2026-09-25 16:40:51 on dist-test-slave-kxl8"
server_key: "168f58862204c9a66cb9bf306f703073"
server_key_iv: "78859457dbc0ac3483cc478c01b26aa1"
server_key_version: "encryptionkey@0"
I20260925 16:40:51.156628 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "43693605520f4d5ab45f3a54f5199cbf"
format_stamp: "Formatted at 2026-09-25 16:40:51 on dist-test-slave-kxl8"
server_key: "168f58862204c9a66cb9bf306f703073"
server_key_iv: "78859457dbc0ac3483cc478c01b26aa1"
server_key_version: "encryptionkey@0"
I20260925 16:40:51.160305 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:40:51.162861 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:51.164108 13996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:40:51.164352 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "43693605520f4d5ab45f3a54f5199cbf"
format_stamp: "Formatted at 2026-09-25 16:40:51 on dist-test-slave-kxl8"
server_key: "168f58862204c9a66cb9bf306f703073"
server_key_iv: "78859457dbc0ac3483cc478c01b26aa1"
server_key_version: "encryptionkey@0"
I20260925 16:40:51.164469 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:51.185117 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:51.187904 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:51.188109 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:51.194967 13996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42177
I20260925 16:40:51.194991 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42177 every 8 connection(s)
I20260925 16:40:51.195881 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:51.198522 14064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:51.203187 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 13996
I20260925 16:40:51.203332 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:40:51.203619 391 external_mini_cluster.cc:1428] Setting key 3ca572ac082ee38c4693951a455a1a59
I20260925 16:40:51.204054 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf: Bootstrap starting.
I20260925 16:40:51.206424 14064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:51.207389 14064 log.cc:824] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:51.209429 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf: No bootstrap required, opened a new log
Sep 25 16:40:51 dist-test-slave-kxl8 krb5kdc[13983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:51.214040 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } }
I20260925 16:40:51.214366 14064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:51.214432 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43693605520f4d5ab45f3a54f5199cbf, State: Initialized, Role: FOLLOWER
I20260925 16:40:51.214921 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } }
I20260925 16:40:51.215070 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:51.215134 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:51.215229 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:51.216140 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } }
I20260925 16:40:51.216485 14064 leader_election.cc:302] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43693605520f4d5ab45f3a54f5199cbf; no voters:
I20260925 16:40:51.216850 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:40:51.216972 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:51.217187 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [term 1 LEADER]: Becoming Leader. State: Replica: 43693605520f4d5ab45f3a54f5199cbf, State: Running, Role: LEADER
I20260925 16:40:51.217564 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } }
I20260925 16:40:51.218176 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:51.204918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.0.97.254:42177)
0925 16:40:51.205392 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:51.205408 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:51.205442 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:40:51.206052 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:51.206068 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:51.206389 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:51.206705 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:51.207590 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:51.208537 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:51.209718 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:51.210232 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:51.210352 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:51.213158 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:51.213163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:51.213176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:51.213205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:51.215047 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:51.215592 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:51.215597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:51.215601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:51.215707 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:51.215999 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:51.216006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:51.216008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:51.216459 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:51.216664 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:51.216954 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 16:40:51.217344 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260925 16:40:51.218225 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:51.219300 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43693605520f4d5ab45f3a54f5199cbf. Latest consensus state: current_term: 1 leader_uuid: "43693605520f4d5ab45f3a54f5199cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } } }
I20260925 16:40:51.219427 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:51.219753 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43693605520f4d5ab45f3a54f5199cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43693605520f4d5ab45f3a54f5199cbf" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42177 } } }
I20260925 16:40:51.219844 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:51.221364 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:40:51.221484 14078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:51.224766 14078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:51.230520 14078 catalog_manager.cc:1347] Generated new cluster ID: 280e8a3e461240338720976f166f92e5
I20260925 16:40:51.230594 14078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:51.253162 14078 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:40:51.254241 14078 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:51.260715 14078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43693605520f4d5ab45f3a54f5199cbf: Generated new TSK 0
I20260925 16:40:51.261463 14078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:40:51.277096 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 13996
2026-09-25T16:40:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1412 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:40:51 dist-test-slave-kxl8 krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:40:51 dist-test-slave-kxl8 krb5kdc[14091](info): set up 2 sockets
Sep 25 16:40:51 dist-test-slave-kxl8 krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260925 16:40:52.309515 391 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:40:52 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:40:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:40:52.432890 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:52.544138 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:52.544425 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:52.544502 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:52.548552 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:52.548632 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:52.548671 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:52.548727 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:52.548763 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:52.553972 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:52.555071 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:52.556118 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:52.564706 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:52.564730 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:52.564705 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:52.564826 14104 server_base.cc:1034] running on GCE node
I20260925 16:40:52.565623 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:52.566817 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:52.567982 14104 hybrid_clock.cc:648] HybridClock initialized: now 1790354452567966 us; error 27 us; skew 500 ppm
Sep 25 16:40:52 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:52.577224 14104 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:52.578214 14104 webserver.cc:460] Webserver started at http://127.0.97.254:39017/ using document root <none> and password file <none>
I20260925 16:40:52.578783 14104 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:52.578873 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:52.579103 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:52.581230 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "b0e1015b3e3b4f1382d2f45610a85190"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "d8560c80d9dac3577d39652dcc6dde8d"
server_key_iv: "94af37011c26d32fc1ee9164940845df"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.581861 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "b0e1015b3e3b4f1382d2f45610a85190"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "d8560c80d9dac3577d39652dcc6dde8d"
server_key_iv: "94af37011c26d32fc1ee9164940845df"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.585575 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:40:52.587970 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:52.589210 14104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:52.589368 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "b0e1015b3e3b4f1382d2f45610a85190"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "d8560c80d9dac3577d39652dcc6dde8d"
server_key_iv: "94af37011c26d32fc1ee9164940845df"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.589481 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:52.604415 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:52.607719 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:52.607924 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:52.614985 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36959
I20260925 16:40:52.615306 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36959 every 8 connection(s)
I20260925 16:40:52.616106 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:52.618760 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:52.619105 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14104
I20260925 16:40:52.619228 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:40:52.619521 391 external_mini_cluster.cc:1428] Setting key f27c26aaf3f0e97d57134f07e647f4a7
I20260925 16:40:52.625062 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap starting.
I20260925 16:40:52.627195 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:52.627895 14172 log.cc:824] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:40:52 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:52.630074 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: No bootstrap required, opened a new log
I20260925 16:40:52.633186 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:52.620893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.97.254:36959)
0925 16:40:52.621399 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:52.621414 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:52.621448 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:40:52.622081 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:52.622097 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:52.622415 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:52.622752 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:52.623536 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.624574 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.625489 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.625962 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.626145 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:52.628647 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:52.628660 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:52.628674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:52.628698 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:52.630482 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:52.630978 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:52.630982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:52.630986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:52.631107 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:52.631414 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:52.631416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:52.631417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:52.631786 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:52.631927 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:52.632209 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 16:40:52.632473 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
I20260925 16:40:52.634492 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:52.634706 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:52.634763 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Initialized, Role: FOLLOWER
I20260925 16:40:52.635246 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:52.635408 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:52.635473 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:52.635568 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:52.636435 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:52.636761 14172 leader_election.cc:302] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e1015b3e3b4f1382d2f45610a85190; no voters:
I20260925 16:40:52.637055 14172 leader_election.cc:288] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:40:52.637179 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:52.637454 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 LEADER]: Becoming Leader. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Running, Role: LEADER
I20260925 16:40:52.637770 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:52.638352 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:52.639420 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e1015b3e3b4f1382d2f45610a85190. Latest consensus state: current_term: 1 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:52.639535 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:52.639833 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:52.639925 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:52.640038 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:52.643220 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:52.649497 14186 catalog_manager.cc:1347] Generated new cluster ID: 973b85691ba3451ca8af6fda7cbd32bc
I20260925 16:40:52.649569 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:52.659686 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:40:52.660888 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:52.674108 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Generated new TSK 0
I20260925 16:40:52.675004 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:40:52.705243 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:52.814790 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:52.815073 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:52.815147 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:52.819134 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:52.819228 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:52.819324 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:40:52.824548 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:52.825623 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:52.826651 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:52.836077 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:52.836077 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:52.836081 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:52.836285 14198 server_base.cc:1034] running on GCE node
I20260925 16:40:52.836702 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:52.837251 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:52.838471 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790354452838458 us; error 24 us; skew 500 ppm
Sep 25 16:40:52 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:52.841439 14198 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:40:52.842211 14198 webserver.cc:460] Webserver started at http://127.0.97.193:45921/ using document root <none> and password file <none>
I20260925 16:40:52.842716 14198 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:52.842772 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:52.842926 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:52.844833 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "ff986f27904e6481565198d8107b74ba"
server_key_iv: "78bbc3888ffa2350ccb9e9920bed0faa"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.845427 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "ff986f27904e6481565198d8107b74ba"
server_key_iv: "78bbc3888ffa2350ccb9e9920bed0faa"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.848889 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:52.851279 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:52.852509 14198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:52.852655 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "ff986f27904e6481565198d8107b74ba"
server_key_iv: "78bbc3888ffa2350ccb9e9920bed0faa"
server_key_version: "encryptionkey@0"
I20260925 16:40:52.852766 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:52.873049 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:52.876365 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:52.876571 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:52.877158 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:52.877987 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:52.878067 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:52.878139 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:52.878193 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:52.887238 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:36361
I20260925 16:40:52.887274 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:36361 every 8 connection(s)
I20260925 16:40:52.888252 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:40:52.891683 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14198
I20260925 16:40:52.891816 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:40:52.892113 391 external_mini_cluster.cc:1428] Setting key d5b2450dba644eab7c7bb2f23a515e90
Sep 25 16:40:52 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:40:52.903255 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:52.890073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:52065 (local address 127.0.97.254:36959)
0925 16:40:52.890209 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:52.890212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:52.890970 (+ 758us) server_negotiation.cc:407] Connection header received
0925 16:40:52.892466 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:52.892470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:52.892529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:52.892603 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:52.893897 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.894425 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.896180 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.896410 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.896495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:52.899537 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:52.899541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:52.899543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:52.899567 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:52.901054 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:52.901642 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:52.901647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:52.901653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:52.901729 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:52.902075 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:52.902078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:52.902080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:52.902335 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:52.902446 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:52.902958 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 16:40:52.903088 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
I20260925 16:40:52.904244 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:52.890388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:52065)
0925 16:40:52.890824 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:52.890864 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:52.892236 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:52.892772 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:52.892779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:52.893125 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:52.893757 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:52.893772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.894548 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.894551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:52.896048 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:52.896062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:52.896568 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:52.896574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:52.896812 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:52.897567 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:52.897595 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:52.899350 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:52.901182 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:52.901188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:52.901206 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:52.901527 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:52.901822 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:52.901826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:52.901828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:52.901972 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:52.902442 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:52.902449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:52.902716 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 16:40:52.902902 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:40:52.903252 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260925 16:40:52.905541 14327 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:52.905812 14327 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:52.906307 14327 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:52.907799 14136 ts_manager.cc:194] Registered new tserver with Master: 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361)
I20260925 16:40:52.908917 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:52065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:40:52.920578 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:53.028977 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:53.029346 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:53.029409 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:53.033355 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:53.033406 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:53.033463 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:40:53.038687 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:53.039819 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:53.040871 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:53.050294 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.050290 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.050384 14332 server_base.cc:1034] running on GCE node
W20260925 16:40:53.050292 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.050899 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:53.051514 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:53.052698 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790354453052677 us; error 34 us; skew 500 ppm
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:53.055511 14332 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:40:53.056411 14332 webserver.cc:460] Webserver started at http://127.0.97.194:41859/ using document root <none> and password file <none>
I20260925 16:40:53.057058 14332 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:53.057145 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:53.057415 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:53.059387 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "93636cbb558d4dbf8b8085f66d906458"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "acb144e256636ffe182a169617e9e544"
server_key_iv: "a93b562ba8784df5e9ba978778019832"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.059947 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "93636cbb558d4dbf8b8085f66d906458"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "acb144e256636ffe182a169617e9e544"
server_key_iv: "a93b562ba8784df5e9ba978778019832"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.063457 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:40:53.065817 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.066972 14332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:40:53.067101 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "93636cbb558d4dbf8b8085f66d906458"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "acb144e256636ffe182a169617e9e544"
server_key_iv: "a93b562ba8784df5e9ba978778019832"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.067209 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:53.097553 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:53.100908 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:53.101114 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:53.101779 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:53.102599 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:53.102679 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.102752 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:53.102795 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.111780 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:35547
I20260925 16:40:53.111855 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:35547 every 8 connection(s)
I20260925 16:40:53.112710 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:40:53.116808 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14332
I20260925 16:40:53.116940 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:40:53.117199 391 external_mini_cluster.cc:1428] Setting key 869b6ec87c4945d432003cbc3dc3cf6e
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:40:53.127677 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.114443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:59027 (local address 127.0.97.254:36959)
0925 16:40:53.114575 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.114579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.115441 (+ 862us) server_negotiation.cc:407] Connection header received
0925 16:40:53.116719 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.116722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.116776 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.116887 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:53.118183 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.118796 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.120544 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.120929 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.121041 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.123963 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:53.123966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:53.123968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:53.123991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:53.125527 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.126101 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.126104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.126105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.126176 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.126578 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.126580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.126581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.126757 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:53.126829 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.127447 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.127547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:40:53.128520 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.114756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:59027)
0925 16:40:53.115296 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.115339 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.116512 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.117066 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.117074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.117511 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:53.118042 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.118061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.118916 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.118920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.120410 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.120425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.121080 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.121083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.121257 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.121970 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:53.121988 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:53.123782 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:53.125638 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.125647 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.125664 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.125999 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.126275 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.126278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.126280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.126472 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.126858 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:53.126865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:53.127191 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.127393 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.127636 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:40:53.130064 14461 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:53.130353 14461 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:53.130873 14461 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:53.132081 14136 ts_manager.cc:194] Registered new tserver with Master: 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
I20260925 16:40:53.132656 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:59027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:40:53.144238 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:53.252514 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:53.252748 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:53.252790 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:53.256870 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:53.256927 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:53.256984 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:40:53.262317 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:53.263299 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:53.264215 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:40:53.273758 14466 server_base.cc:1034] running on GCE node
W20260925 16:40:53.273722 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.273870 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.273718 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.274358 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:53.275004 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:53.276191 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790354453276179 us; error 26 us; skew 500 ppm
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:53.278846 14466 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:40:53.279609 14466 webserver.cc:460] Webserver started at http://127.0.97.195:36497/ using document root <none> and password file <none>
I20260925 16:40:53.280100 14466 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:53.280162 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:53.280318 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:40:53.282310 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "979786bf10ad4c7f8521761a5c14093a"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "f5754fee42ec9c037e1fa597ddc0e17c"
server_key_iv: "39b4eb4bb421481bb7940cdd26e259de"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.282816 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "979786bf10ad4c7f8521761a5c14093a"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "f5754fee42ec9c037e1fa597ddc0e17c"
server_key_iv: "39b4eb4bb421481bb7940cdd26e259de"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.286422 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 16:40:53.288743 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.290477 14466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:40:53.290583 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "979786bf10ad4c7f8521761a5c14093a"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "f5754fee42ec9c037e1fa597ddc0e17c"
server_key_iv: "39b4eb4bb421481bb7940cdd26e259de"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.290699 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:53.306577 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:53.309804 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:53.310004 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:53.310624 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:53.311475 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:53.311546 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.311619 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:53.311663 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.324244 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:45815 every 8 connection(s)
I20260925 16:40:53.324226 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:45815
I20260925 16:40:53.325958 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:40:53.330181 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14466
I20260925 16:40:53.330333 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:40:53.330637 391 external_mini_cluster.cc:1428] Setting key df5f65c468c6b62954358fbdf7eacb56
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:53.346326 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.328105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:48629 (local address 127.0.97.254:36959)
0925 16:40:53.328305 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.328311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.331549 (+ 3238us) server_negotiation.cc:407] Connection header received
0925 16:40:53.333369 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.333372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.333440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.333516 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:53.335118 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.336100 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.337913 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.338283 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.338385 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.341806 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:53.341809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:53.341812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:53.341840 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:53.343806 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.344506 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.344510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.344512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.344613 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.344969 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.344973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.344974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.345231 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:53.345373 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.346037 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.346166 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:40:53.347523 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.328549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:48629)
0925 16:40:53.331393 (+ 2844us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.331452 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.333078 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.333705 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.333717 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.334270 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:53.334972 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.334993 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.336238 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.336240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.337738 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.337761 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.338595 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.338599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.338909 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.339610 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:53.339632 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:53.341579 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:53.343941 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.343950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.343969 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.344369 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.344700 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.344706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.344709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.344857 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.345371 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:53.345380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:53.345742 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.345970 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.346341 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":159,"threads_started":1}
I20260925 16:40:53.348706 14595 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:53.348958 14595 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:53.349570 14595 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:53.350791 14136 ts_manager.cc:194] Registered new tserver with Master: 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815)
I20260925 16:40:53.351675 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:48629
I20260925 16:40:53.358078 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:40:53.368494 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.360079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.97.254:36959)
0925 16:40:53.360202 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.360207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.360442 (+ 235us) server_negotiation.cc:407] Connection header received
0925 16:40:53.360528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.360530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.360566 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.360654 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:53.361384 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.361899 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.363259 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.363468 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.363601 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.365466 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:53.365469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:53.365471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:53.365492 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:53.366994 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.367385 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.367387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.367388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.367458 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.367694 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.367696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.367697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.367898 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:53.368000 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.368200 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.368288 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:40:53.371567 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:40:53.374538 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:40:53.388199 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.382623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:35547 (local address 127.0.0.1:51796)
0925 16:40:53.382961 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.383003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.383118 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.383865 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.383874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.383897 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:53.384024 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.384031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.385489 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.385492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.386551 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.386560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.387644 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.387647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.387774 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.387823 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.387928 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.388038 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260925 16:40:53.388222 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.382750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.0.97.194:35547)
0925 16:40:53.383080 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.383091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.383122 (+ 31us) server_negotiation.cc:407] Connection header received
0925 16:40:53.383248 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.383255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.383463 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.383643 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:53.384150 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.385211 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.386680 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.387525 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.387630 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.387769 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.387979 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.388076 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":149,"threads_started":1}
I20260925 16:40:53.388909 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.383368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.0.1:42138)
0925 16:40:53.383598 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.383612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.383689 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.384472 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.384476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.384497 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:53.384606 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.384614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.386012 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.386015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.387243 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.387258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.388526 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.388530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.388617 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.388638 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.388716 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.388788 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":47,"thread_start_us":91,"threads_started":1}
I20260925 16:40:53.389024 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.383507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.97.193:36361)
0925 16:40:53.384028 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.384041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.384059 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:40:53.384145 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.384150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.384327 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.384479 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:53.384745 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.385872 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.387396 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.388219 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.388328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.388463 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.388774 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.388872 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":62,"threads_started":1}
I20260925 16:40:53.389945 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.383664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.0.1:49652)
0925 16:40:53.384705 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.384718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.384819 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.385247 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.385249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.385269 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:53.385444 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.385453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.386628 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.386633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.387688 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.387696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.389584 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.389587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.389657 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.389672 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.389762 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.389827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260925 16:40:53.390256 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.383981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.0.97.195:45815)
0925 16:40:53.384296 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.384307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.384824 (+ 517us) server_negotiation.cc:407] Connection header received
0925 16:40:53.384911 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.384916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.385097 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.385243 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:53.385546 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.386489 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.387820 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.389252 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.389399 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.389525 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.389792 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.390125 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":143,"threads_started":1}
I20260925 16:40:53.391405 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 (DEFAULT_TABLE table=test-table [id=f10e7a265fb446d5a9477172c4a87c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:53.391405 14261 tablet_service.cc:1467] Processing CreateTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 (DEFAULT_TABLE table=test-table [id=f10e7a265fb446d5a9477172c4a87c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:53.392522 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085a21e7ae204ae7b6969127baec7ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:53.392522 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085a21e7ae204ae7b6969127baec7ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:53.395987 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 (DEFAULT_TABLE table=test-table [id=f10e7a265fb446d5a9477172c4a87c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:53.396940 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085a21e7ae204ae7b6969127baec7ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:53.400420 14613 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Bootstrap starting.
I20260925 16:40:53.400869 14612 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap starting.
I20260925 16:40:53.402855 14613 tablet_bootstrap.cc:654] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:53.403074 14612 tablet_bootstrap.cc:654] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:53.403442 14614 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Bootstrap starting.
I20260925 16:40:53.403739 14613 log.cc:824] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:53.404492 14612 log.cc:824] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:53.405511 14614 tablet_bootstrap.cc:654] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:53.405891 14613 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: No bootstrap required, opened a new log
I20260925 16:40:53.406106 14613 ts_tablet_manager.cc:1397] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:40:53.406363 14612 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: No bootstrap required, opened a new log
I20260925 16:40:53.406561 14612 ts_tablet_manager.cc:1397] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:40:53.406652 14614 log.cc:824] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:53.408936 14614 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: No bootstrap required, opened a new log
I20260925 16:40:53.409103 14614 ts_tablet_manager.cc:1397] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:40:53.409857 14613 raft_consensus.cc:348] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.409940 14612 raft_consensus.cc:348] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.410125 14613 raft_consensus.cc:374] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:53.410162 14612 raft_consensus.cc:374] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:53.410187 14613 raft_consensus.cc:729] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93636cbb558d4dbf8b8085f66d906458, State: Initialized, Role: FOLLOWER
I20260925 16:40:53.410207 14612 raft_consensus.cc:729] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb1d7785c0e4101b96b3e9ed9c39a28, State: Initialized, Role: FOLLOWER
I20260925 16:40:53.410600 14612 consensus_queue.cc:260] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.410645 14613 consensus_queue.cc:260] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.411178 14327 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
I20260925 16:40:53.411598 14613 ts_tablet_manager.cc:1428] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:40:53.411922 14461 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
I20260925 16:40:53.412460 14612 ts_tablet_manager.cc:1428] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:40:53.414318 14614 raft_consensus.cc:348] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.414592 14614 raft_consensus.cc:374] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:53.414649 14614 raft_consensus.cc:729] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979786bf10ad4c7f8521761a5c14093a, State: Initialized, Role: FOLLOWER
I20260925 16:40:53.415105 14614 consensus_queue.cc:260] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.416115 14614 ts_tablet_manager.cc:1428] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260925 16:40:53.416440 14595 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
I20260925 16:40:53.450840 14619 raft_consensus.cc:482] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:53.451062 14619 raft_consensus.cc:504] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.452204 14619 leader_election.cc:288] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361), 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815)
I20260925 16:40:53.455778 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.452449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.97.194:37455)
0925 16:40:53.452582 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.452600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.452691 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.452968 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.452971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.452989 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:53.453114 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.453119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.453927 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.453932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.454710 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.454719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.455348 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.455352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.455454 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.455481 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.455568 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.455638 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 16:40:53.455778 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.452586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37455 (local address 127.0.97.193:36361)
0925 16:40:53.452720 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.452725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.452740 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:40:53.452797 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.452800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.452855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.452942 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:53.453247 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.453814 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.454831 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.455232 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.455357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.455408 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.455584 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.455658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:40:53.456144 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.452726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.97.194:47021)
0925 16:40:53.453137 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.453147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.453267 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.453526 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.453529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.453544 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:53.453662 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.453667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.454342 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.454348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.455097 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.455104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.455725 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.455729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.455836 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.455857 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.455931 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.456021 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":145,"threads_started":1}
I20260925 16:40:53.456152 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.452852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:47021 (local address 127.0.97.195:45815)
0925 16:40:53.452999 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.453004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.453220 (+ 216us) server_negotiation.cc:407] Connection header received
0925 16:40:53.453389 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.453392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.453434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.453516 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:53.453771 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.454238 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.455217 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.455609 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.455735 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.455790 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.455956 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.456043 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:40:53.456506 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085a21e7ae204ae7b6969127baec7ae7" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" is_pre_election: true
I20260925 16:40:53.456856 14281 raft_consensus.cc:2454] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 0.
I20260925 16:40:53.456868 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085a21e7ae204ae7b6969127baec7ae7" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979786bf10ad4c7f8521761a5c14093a" is_pre_election: true
I20260925 16:40:53.457145 14549 raft_consensus.cc:2454] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 0.
I20260925 16:40:53.457374 14351 leader_election.cc:302] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28, 93636cbb558d4dbf8b8085f66d906458; no voters:
I20260925 16:40:53.457675 14619 raft_consensus.cc:2781] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:40:53.457757 14619 raft_consensus.cc:482] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:40:53.457811 14619 raft_consensus.cc:3037] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:53.459000 14619 raft_consensus.cc:504] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.459345 14619 leader_election.cc:288] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 election: Requested vote from peers 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361), 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815)
I20260925 16:40:53.459759 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085a21e7ae204ae7b6969127baec7ae7" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979786bf10ad4c7f8521761a5c14093a"
I20260925 16:40:53.459792 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085a21e7ae204ae7b6969127baec7ae7" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
I20260925 16:40:53.459914 14549 raft_consensus.cc:3037] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:53.459937 14281 raft_consensus.cc:3037] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:53.460917 14549 raft_consensus.cc:2454] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 1.
I20260925 16:40:53.461139 14281 raft_consensus.cc:2454] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 1.
I20260925 16:40:53.461197 14351 leader_election.cc:302] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a; no voters:
I20260925 16:40:53.461401 14619 raft_consensus.cc:2781] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:53.461676 14619 raft_consensus.cc:686] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 1 LEADER]: Becoming Leader. State: Replica: 93636cbb558d4dbf8b8085f66d906458, State: Running, Role: LEADER
I20260925 16:40:53.462049 14619 consensus_queue.cc:237] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.465206 14136 catalog_manager.cc:5495] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "93636cbb558d4dbf8b8085f66d906458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } health_report { overall_health: UNKNOWN } } }
I20260925 16:40:53.489933 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.486666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.97.194:35547)
0925 16:40:53.486797 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.486801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.486816 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:40:53.486929 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.486931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.486988 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.487079 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:40:53.487293 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.487784 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.488586 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.488784 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.488855 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.489103 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:40:53.489549 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:40:53.489640 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.489756 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.489806 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:40:53.495795 14549 raft_consensus.cc:1261] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Refusing update from remote peer 93636cbb558d4dbf8b8085f66d906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:53.495793 14281 raft_consensus.cc:1261] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Refusing update from remote peer 93636cbb558d4dbf8b8085f66d906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:53.496553 14622 consensus_queue.cc:1035] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:53.496755 14619 consensus_queue.cc:1035] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:53.504639 14627 mvcc.cc:205] Tried to move back new op lower bound from 7333291841511612416 to 7333291841383456768. Current Snapshot: MvccSnapshot[applied={T|T < 7333291841511612416}]
I20260925 16:40:53.514235 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:53.514426 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:53.516808 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Sending DeleteTablet for 3 replicas of tablet 085a21e7ae204ae7b6969127baec7ae7
I20260925 16:40:53.517612 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:51796
I20260925 16:40:53.517673 14261 tablet_service.cc:1514] Processing DeleteTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260925 16:40:53.517817 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:49652
I20260925 16:40:53.518338 14637 tablet_replica.cc:330] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: stopping tablet replica
I20260925 16:40:53.518503 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14198
I20260925 16:40:53.519697 14637 raft_consensus.cc:2229] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:40:53.519716 14636 tablet_replica.cc:330] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: stopping tablet replica
I20260925 16:40:53.520032 14637 raft_consensus.cc:2258] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:53.520627 14636 raft_consensus.cc:2229] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:40:53.521092 14636 raft_consensus.cc:2258] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:53.521545 14637 ts_tablet_manager.cc:1905] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:53.522505 14636 ts_tablet_manager.cc:1905] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:53.524992 14637 ts_tablet_manager.cc:1918] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:53.525099 14637 log.cc:1196] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/085a21e7ae204ae7b6969127baec7ae7
I20260925 16:40:53.525308 14636 ts_tablet_manager.cc:1918] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:53.525462 14636 log.cc:1196] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/085a21e7ae204ae7b6969127baec7ae7
I20260925 16:40:53.525483 14637 ts_tablet_manager.cc:1939] T 085a21e7ae204ae7b6969127baec7ae7 P 979786bf10ad4c7f8521761a5c14093a: Deleting consensus metadata
I20260925 16:40:53.526005 14636 ts_tablet_manager.cc:1939] T 085a21e7ae204ae7b6969127baec7ae7 P 93636cbb558d4dbf8b8085f66d906458: Deleting consensus metadata
W20260925 16:40:53.526656 14123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:36361 (error 108)
I20260925 16:40:53.527145 14123 catalog_manager.cc:4841] TS 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): tablet 085a21e7ae204ae7b6969127baec7ae7 (table test-table [id=f10e7a265fb446d5a9477172c4a87c29]) successfully deleted
I20260925 16:40:53.527357 14123 catalog_manager.cc:4841] TS 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): tablet 085a21e7ae204ae7b6969127baec7ae7 (table test-table [id=f10e7a265fb446d5a9477172c4a87c29]) successfully deleted
I20260925 16:40:53.527690 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14332
I20260925 16:40:53.527922 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.527624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.0.1:42152)
0925 16:40:53.527775 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.527845 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 16:40:53.528861 14123 catalog_manager.cc:4584] TS 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 085a21e7ae204ae7b6969127baec7ae7: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36361: connect: Connection refused (error 111)
I20260925 16:40:53.532888 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14466
I20260925 16:40:53.538311 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14104
I20260925 16:40:53.544827 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--webserver_interface=127.0.97.254
--webserver_port=39017
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:53.653973 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:53.654277 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:53.654353 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:53.658403 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:53.658483 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:53.658520 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:53.658581 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:53.658618 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:53.664088 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:53.665172 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:53.666242 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:53.679126 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.679325 14639 server_base.cc:1034] running on GCE node
W20260925 16:40:53.679129 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.679126 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.680056 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:53.681265 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:53.682492 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790354453682454 us; error 49 us; skew 500 ppm
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:53.685139 14639 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:53.686116 14639 webserver.cc:460] Webserver started at http://127.0.97.254:39017/ using document root <none> and password file <none>
I20260925 16:40:53.686695 14639 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:53.686803 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:53.690595 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 16:40:53.692687 14655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.693977 14639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:53.694118 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "b0e1015b3e3b4f1382d2f45610a85190"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "d8560c80d9dac3577d39652dcc6dde8d"
server_key_iv: "94af37011c26d32fc1ee9164940845df"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.694705 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:53.721153 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:53.724547 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:53.724764 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:53.732085 14639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36959
I20260925 16:40:53.732398 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36959 every 8 connection(s)
I20260925 16:40:53.733204 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:53.737617 14708 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:40:53.739305 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap starting.
I20260925 16:40:53.741158 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14639
I20260925 16:40:53.741626 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:36361
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=45921
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:53.756247 14708 log.cc:824] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:53.765582 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:40:53.765969 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap complete.
I20260925 16:40:53.770092 14708 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:53.770495 14708 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Initialized, Role: FOLLOWER
I20260925 16:40:53.770982 14708 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:53.771149 14708 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:53.771239 14708 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:53.771361 14708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:40:53.773218 14708 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:53.773581 14708 leader_election.cc:302] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e1015b3e3b4f1382d2f45610a85190; no voters:
I20260925 16:40:53.773917 14708 leader_election.cc:288] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:40:53.774003 14712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:40:53.774205 14712 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 LEADER]: Becoming Leader. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Running, Role: LEADER
I20260925 16:40:53.774489 14712 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:53.775167 14708 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:53.775843 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:53.775961 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:53.776513 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e1015b3e3b4f1382d2f45610a85190. Latest consensus state: current_term: 2 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:53.776657 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:53.776911 14719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:53.781737 14719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:53.782342 14719 catalog_manager.cc:1259] Loaded cluster ID: 973b85691ba3451ca8af6fda7cbd32bc
I20260925 16:40:53.782398 14719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:40:53.783653 14719 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 16:40:53.788340 14719 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 16:40:53.793519 14719 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:40:53.794207 14719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Loaded TSK: 0
I20260925 16:40:53.795365 14719 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:40:53.855913 14710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:53.856156 14710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:53.856200 14710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:53.860455 14710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:53.860512 14710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:53.860569 14710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:40:53.865631 14710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:36361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=45921
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:53.866585 14710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:53.867563 14710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:53.877154 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.877138 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:53.877138 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:53.877739 14710 server_base.cc:1034] running on GCE node
I20260925 16:40:53.878170 14710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:53.878882 14710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:53.880084 14710 hybrid_clock.cc:648] HybridClock initialized: now 1790354453880058 us; error 35 us; skew 500 ppm
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:53.883416 14710 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:40:53.884528 14710 webserver.cc:460] Webserver started at http://127.0.97.193:45921/ using document root <none> and password file <none>
I20260925 16:40:53.885217 14710 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:53.885352 14710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:53.890345 14710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 16:40:53.892897 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:53.894403 14710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:40:53.894527 14710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "ff986f27904e6481565198d8107b74ba"
server_key_iv: "78bbc3888ffa2350ccb9e9920bed0faa"
server_key_version: "encryptionkey@0"
I20260925 16:40:53.895223 14710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:53.912281 14710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:53.915979 14710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:53.916196 14710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:53.916882 14710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:53.918202 14752 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:40:53.922444 14710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:40:53.922518 14710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 16:40:53.922578 14710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:40:53.924606 14710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:40:53.924659 14710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:40:53.925015 14752 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap starting.
I20260925 16:40:53.934513 14710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:36361
I20260925 16:40:53.934885 14859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:36361 every 8 connection(s)
I20260925 16:40:53.935563 14710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:40:53.937906 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14710
I20260925 16:40:53.938351 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:35547
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=41859
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:53.944852 14752 log.cc:824] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:53.954002 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.937553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:55901)
0925 16:40:53.938694 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.938772 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.939875 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.941315 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.941328 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.941874 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:53.942499 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.942522 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.944666 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.944670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.945790 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.945799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.946541 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.946545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.946733 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.947359 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:53.947376 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:53.948820 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:53.950917 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.950922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.950934 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.951204 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.951560 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.951564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.951566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.951700 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.952288 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:53.952294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:53.952527 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.952749 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.953078 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":147,"threads_started":1}
I20260925 16:40:53.953989 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.937553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:55901 (local address 127.0.97.254:36959)
0925 16:40:53.938029 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.938047 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.938935 (+ 888us) server_negotiation.cc:407] Connection header received
0925 16:40:53.940392 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.940409 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.940837 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.941118 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:53.942711 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.944469 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.945925 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.946387 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.946574 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.949028 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:53.949034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:53.949045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:53.949072 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:53.950788 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.951344 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.951349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.951353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.951461 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.951807 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.951810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.951812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.952151 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:53.952302 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.952886 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.953142 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":105,"threads_started":1}
I20260925 16:40:53.954166 14752 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:40:53.954598 14752 tablet_bootstrap.cc:492] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap complete.
I20260925 16:40:53.956030 14752 ts_tablet_manager.cc:1397] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.012s
I20260925 16:40:53.956593 14860 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:53.956844 14860 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:53.957383 14860 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:53.959128 14752 raft_consensus.cc:348] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.959646 14752 raft_consensus.cc:729] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb1d7785c0e4101b96b3e9ed9c39a28, State: Initialized, Role: FOLLOWER
I20260925 16:40:53.959838 14672 ts_manager.cc:194] Registered new tserver with Master: 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361)
I20260925 16:40:53.960129 14752 consensus_queue.cc:260] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } } peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } }
I20260925 16:40:53.961182 14752 ts_tablet_manager.cc:1428] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 16:40:53.963488 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:55901
I20260925 16:40:53.965864 14860 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
Sep 25 16:40:53 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:40:53.973472 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.962791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.0.1:42164)
0925 16:40:53.963474 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:53.963525 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:53.963685 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:53.964390 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:53.964400 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:53.964844 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:53.964970 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.964977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.966225 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.966228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.966940 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:53.966948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.967636 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.967640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:53.967748 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.968282 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:53.968297 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:53.969867 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:53.971701 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.971710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.971716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.971947 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.972247 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:53.972250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:53.972252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:53.972343 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:53.972791 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:53.972797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:53.972921 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 16:40:53.973061 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 16:40:53.973255 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":102,"threads_started":1}
I20260925 16:40:53.973475 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:53.962891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.0.97.193:36361)
0925 16:40:53.963488 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:53.963501 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:53.963819 (+ 318us) server_negotiation.cc:407] Connection header received
0925 16:40:53.963892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:53.963897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:53.964089 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:53.964215 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:53.965106 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.966122 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.967060 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:53.967517 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:53.967621 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:53.970014 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:53.970022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:53.970032 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:53.970052 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:53.971587 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.972064 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.972071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.972073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.972153 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:53.972435 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:53.972439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:53.972441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:53.972672 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:53.972812 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:53.973137 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 16:40:53.973334 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":100,"threads_started":1}
I20260925 16:40:53.974408 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 085a21e7ae204ae7b6969127baec7ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42164
I20260925 16:40:53.975255 14869 tablet_replica.cc:330] stopping tablet replica
I20260925 16:40:53.975593 14869 raft_consensus.cc:2229] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:40:53.975838 14869 raft_consensus.cc:2258] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:53.976902 14869 ts_tablet_manager.cc:1905] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:53.980280 14869 ts_tablet_manager.cc:1918] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:53.980360 14869 log.cc:1196] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/085a21e7ae204ae7b6969127baec7ae7
I20260925 16:40:53.980616 14869 ts_tablet_manager.cc:1939] T 085a21e7ae204ae7b6969127baec7ae7 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Deleting consensus metadata
I20260925 16:40:53.982332 14659 catalog_manager.cc:4841] TS 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361): tablet 085a21e7ae204ae7b6969127baec7ae7 (table test-table [id=f10e7a265fb446d5a9477172c4a87c29]) successfully deleted
W20260925 16:40:54.059116 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:54.059403 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:54.059473 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:54.063481 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:54.063575 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:54.063669 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:40:54.068881 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:35547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=41859
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:54.069963 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:54.070992 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:54.080754 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:54.080754 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:54.080794 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:54.081166 14864 server_base.cc:1034] running on GCE node
I20260925 16:40:54.081614 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:54.082408 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:54.083683 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790354454083669 us; error 28 us; skew 500 ppm
Sep 25 16:40:54 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:54.086668 14864 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:40:54.087530 14864 webserver.cc:460] Webserver started at http://127.0.97.194:41859/ using document root <none> and password file <none>
I20260925 16:40:54.088101 14864 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:54.088203 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:54.091949 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:54.094025 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.095191 14864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:54.095325 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "93636cbb558d4dbf8b8085f66d906458"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "acb144e256636ffe182a169617e9e544"
server_key_iv: "a93b562ba8784df5e9ba978778019832"
server_key_version: "encryptionkey@0"
I20260925 16:40:54.095875 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:54.108913 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:54.111687 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:54.111891 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:54.112463 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:54.113351 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:54.113415 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.113477 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:54.113529 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.122154 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:35547
I20260925 16:40:54.122175 14997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:35547 every 8 connection(s)
I20260925 16:40:54.123142 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:40:54.125854 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 14864
I20260925 16:40:54.126269 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:45815
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=36497
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:40:54 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:54.138166 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.124934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:59901 (local address 127.0.97.254:36959)
0925 16:40:54.125104 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.125110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.125931 (+ 821us) server_negotiation.cc:407] Connection header received
0925 16:40:54.127328 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.127331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.127398 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.127487 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:54.129155 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.130074 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.131227 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.131678 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.131813 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.134271 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:54.134279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:54.134284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:54.134320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:54.136117 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.136597 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.136604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.136609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.136719 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.137041 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.137046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.137049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.137331 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:54.137431 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.137863 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.137995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:40:54.138881 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.125357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:59901)
0925 16:40:54.125787 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.125828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.127081 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.127811 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.127823 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.128358 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:54.129000 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.129015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.130219 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.130221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.131102 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.131110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.131830 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.131832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.131964 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.132649 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:54.132665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:54.134106 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:54.136229 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:54.136233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:54.136244 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:54.136495 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:54.136819 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:54.136821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:54.136822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:54.136943 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:54.137432 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:54.137436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:54.137646 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.137800 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.138026 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260925 16:40:54.139987 14998 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:54.140241 14998 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:54.140777 14998 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:54.141851 14672 ts_manager.cc:194] Registered new tserver with Master: 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
I20260925 16:40:54.142397 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:59901
W20260925 16:40:54.241346 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:54.241631 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:54.241701 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:54.245761 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:54.245831 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:54.245890 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:40:54.251091 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:45815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=36497
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:54.252161 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:54.253187 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:54.263010 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:54.263054 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:54.263314 15001 server_base.cc:1034] running on GCE node
W20260925 16:40:54.263054 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:54.263716 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:54.264387 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:54.265561 15001 hybrid_clock.cc:648] HybridClock initialized: now 1790354454265530 us; error 47 us; skew 500 ppm
Sep 25 16:40:54 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:54.268229 15001 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:40:54.269026 15001 webserver.cc:460] Webserver started at http://127.0.97.195:36497/ using document root <none> and password file <none>
I20260925 16:40:54.269608 15001 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:54.269681 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:54.273541 15001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 16:40:54.275548 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.277154 15001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:40:54.277345 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "979786bf10ad4c7f8521761a5c14093a"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "f5754fee42ec9c037e1fa597ddc0e17c"
server_key_iv: "39b4eb4bb421481bb7940cdd26e259de"
server_key_version: "encryptionkey@0"
I20260925 16:40:54.277907 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:54.294806 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:54.297863 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:54.298053 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:54.298689 15001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:54.299590 15001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:54.299636 15001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.299697 15001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:54.299728 15001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:54.308481 15001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:45815
I20260925 16:40:54.308506 15129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:45815 every 8 connection(s)
I20260925 16:40:54.309484 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:40:54.313081 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15001
Sep 25 16:40:54 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:40:54.322315 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.313732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.0.97.254:36959)
0925 16:40:54.314094 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.314098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.314401 (+ 303us) server_negotiation.cc:407] Connection header received
0925 16:40:54.314499 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.314502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.314552 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.314652 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:54.315427 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.315920 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.316884 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.317233 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.317393 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.318789 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:54.318793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:54.318796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:54.318820 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:54.320617 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.321049 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.321052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.321054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.321137 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.321442 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.321446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.321448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.321625 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:54.321835 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.321983 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.322099 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":67,"threads_started":1}
I20260925 16:40:54.323807 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.311199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:51145 (local address 127.0.97.254:36959)
0925 16:40:54.311336 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.311340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.312257 (+ 917us) server_negotiation.cc:407] Connection header received
0925 16:40:54.313871 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.313874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.313936 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.314091 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:54.315323 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.315889 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.317201 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.317541 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.317654 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.320199 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:54.320203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:54.320205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:54.320244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:54.321741 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.322315 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.322319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.322321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.322406 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:54.322811 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:54.322815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:54.322816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:54.322979 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:54.323086 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.323617 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.323706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:40:54.324700 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.311565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:51145)
0925 16:40:54.312077 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.312123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.313620 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.314181 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.314188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.314678 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:54.315160 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.315173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.316115 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.316119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.317080 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.317091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.317713 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.317722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.317942 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.318509 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:54.318524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:54.320040 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:54.321873 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:54.321878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:54.321892 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:54.322136 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:54.322502 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:54.322505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:54.322507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:54.322691 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:54.323086 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:54.323094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:54.323371 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.323576 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.323897 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260925 16:40:54.325731 15130 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36959
I20260925 16:40:54.325942 15130 heartbeater.cc:464] Registering TS with master...
I20260925 16:40:54.326367 15130 heartbeater.cc:511] Master 127.0.97.254:36959 requested a full tablet report, sending...
I20260925 16:40:54.327309 14672 ts_manager.cc:194] Registered new tserver with Master: 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815)
I20260925 16:40:54.327843 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:51145
I20260925 16:40:54.335566 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:40:54.339427 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.336150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.97.254:36959)
0925 16:40:54.336280 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.336285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.336461 (+ 176us) server_negotiation.cc:407] Connection header received
0925 16:40:54.336574 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.336576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.336664 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.336746 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:40:54.336892 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.337432 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.338223 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.338424 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.338496 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.338744 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:40:54.339093 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:40:54.339178 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.339231 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.339276 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:40:54.340008 14672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:40:54.343232 14672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:40:54.350775 14794 tablet_service.cc:1467] Processing CreateTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 (DEFAULT_TABLE table=test-table [id=b24ebfa07949434c8ff6e5338ef8960e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:54.351423 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7117563ba79e44a0b381b9a154a3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:54.353889 14752 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap starting.
I20260925 16:40:54.354768 14752 tablet_bootstrap.cc:654] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:54.355497 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.349349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.0.1:49668)
0925 16:40:54.349513 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.349529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.349603 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.350557 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.350560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.350579 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.350673 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.350679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.352325 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.352328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.353750 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.353756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.355007 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.355011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.355116 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.355167 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.355223 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.355329 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":140}
I20260925 16:40:54.355573 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.349961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.97.194:35547)
0925 16:40:54.350374 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.350388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.350406 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:40:54.350465 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.350469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.350677 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.350835 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.350999 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.351980 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.353886 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.354869 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.354986 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.355131 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.355335 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.355441 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
I20260925 16:40:54.355898 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.349563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.97.195:45815)
0925 16:40:54.350043 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.350058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.350082 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:40:54.350159 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.350164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.350394 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.350559 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.350809 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.352196 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.353886 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.354869 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.355453 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.355587 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.355638 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.355731 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":161,"threads_started":1}
I20260925 16:40:54.356211 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.349779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:35547 (local address 127.0.0.1:51810)
0925 16:40:54.350011 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.350024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.350112 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.350814 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.350817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.350833 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.350912 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.350917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.352240 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.352244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.353741 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.353752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.354980 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.354984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.355111 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.355191 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.355293 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.355366 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":100,"thread_start_us":107,"threads_started":1}
I20260925 16:40:54.356501 14752 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: No bootstrap required, opened a new log
I20260925 16:40:54.356591 14752 ts_tablet_manager.cc:1397] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:40:54.357055 14752 raft_consensus.cc:348] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.357196 14752 raft_consensus.cc:374] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:54.357245 14752 raft_consensus.cc:729] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb1d7785c0e4101b96b3e9ed9c39a28, State: Initialized, Role: FOLLOWER
I20260925 16:40:54.357383 14752 consensus_queue.cc:260] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.357659 14752 ts_tablet_manager.cc:1428] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:40:54.359081 15064 tablet_service.cc:1467] Processing CreateTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 (DEFAULT_TABLE table=test-table [id=b24ebfa07949434c8ff6e5338ef8960e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:54.359077 14932 tablet_service.cc:1467] Processing CreateTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 (DEFAULT_TABLE table=test-table [id=b24ebfa07949434c8ff6e5338ef8960e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:40:54.360180 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7117563ba79e44a0b381b9a154a3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:54.360175 14932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7117563ba79e44a0b381b9a154a3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:40:54.365154 14866 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:54.365340 14866 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.366331 14866 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
I20260925 16:40:54.369673 15144 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Bootstrap starting.
I20260925 16:40:54.370935 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.367090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:59049 (local address 127.0.97.195:45815)
0925 16:40:54.367196 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.367200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.367214 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:40:54.367314 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.367317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.367364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.367848 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.367946 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.368668 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.369998 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.370365 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.370479 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.370526 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.370737 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.370810 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:40:54.370960 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.366478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.97.193:59049)
0925 16:40:54.367094 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.367110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.367201 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.367482 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.367485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.367502 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.367620 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.367629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.368795 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.368799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.369882 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.369890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.370480 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.370483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.370590 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.370634 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.370724 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.370803 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":2}
I20260925 16:40:54.372040 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.366728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:35547 (local address 127.0.97.193:33601)
0925 16:40:54.367933 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.367945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.368028 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.368356 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.368359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.368374 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.368493 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.368498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.369445 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.369449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.370680 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.370692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.371538 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.371542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.371645 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.371670 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.371777 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.371847 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":45,"thread_start_us":108,"threads_started":1}
I20260925 16:40:54.372079 15144 tablet_bootstrap.cc:654] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:54.372453 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.366817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:33601 (local address 127.0.97.194:35547)
0925 16:40:54.366935 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.366942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.368105 (+ 1163us) server_negotiation.cc:407] Connection header received
0925 16:40:54.368151 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.368154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.368196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.368267 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.368597 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.369330 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.370832 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.371415 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.371531 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.371574 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.372271 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.372344 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:40:54.373001 15144 log.cc:824] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:54.371573 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979786bf10ad4c7f8521761a5c14093a" is_pre_election: true
I20260925 16:40:54.373816 15145 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Bootstrap starting.
I20260925 16:40:54.372943 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93636cbb558d4dbf8b8085f66d906458" is_pre_election: true
W20260925 16:40:54.374239 14749 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): Illegal state: must be running to vote when last-logged opid is not known
W20260925 16:40:54.374758 14749 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): Illegal state: must be running to vote when last-logged opid is not known
I20260925 16:40:54.374866 14749 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28; no voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a
I20260925 16:40:54.375263 14866 raft_consensus.cc:2726] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 16:40:54.375713 15144 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: No bootstrap required, opened a new log
I20260925 16:40:54.375910 15144 ts_tablet_manager.cc:1397] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:40:54.376380 15145 tablet_bootstrap.cc:654] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Neither blocks nor log segments found. Creating new log.
I20260925 16:40:54.377246 15145 log.cc:824] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:54.378785 15144 raft_consensus.cc:348] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.379009 15144 raft_consensus.cc:374] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:54.379068 15144 raft_consensus.cc:729] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979786bf10ad4c7f8521761a5c14093a, State: Initialized, Role: FOLLOWER
I20260925 16:40:54.379256 15145 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: No bootstrap required, opened a new log
I20260925 16:40:54.379428 15145 ts_tablet_manager.cc:1397] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:40:54.379521 15144 consensus_queue.cc:260] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.380143 15130 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
I20260925 16:40:54.380539 15144 ts_tablet_manager.cc:1428] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:40:54.382135 15145 raft_consensus.cc:348] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.382320 15145 raft_consensus.cc:374] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:40:54.382395 15145 raft_consensus.cc:729] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93636cbb558d4dbf8b8085f66d906458, State: Initialized, Role: FOLLOWER
I20260925 16:40:54.382732 15145 consensus_queue.cc:260] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.383695 14998 heartbeater.cc:503] Master 127.0.97.254:36959 was elected leader, sending a full tablet report...
I20260925 16:40:54.383719 15145 ts_tablet_manager.cc:1428] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260925 16:40:54.436744 14861 tablet.cc:2369] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:40:54.560397 15131 tablet.cc:2369] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:40:54.625008 14999 tablet.cc:2369] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:40:54.797895 15149 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:54.798202 15149 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.799434 15149 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361)
I20260925 16:40:54.804235 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.799826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:57769 (local address 127.0.97.195:45815)
0925 16:40:54.799984 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.799989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.800165 (+ 176us) server_negotiation.cc:407] Connection header received
0925 16:40:54.800316 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.800322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.800376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.800456 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.800814 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.801543 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803077 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.803676 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803798 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.803854 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.804034 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.804111 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:40:54.804237 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.799736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.97.194:57769)
0925 16:40:54.800013 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.800028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.800207 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.800507 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.800511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.800538 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.800698 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.800705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.801649 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.801654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.802954 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.802967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.803791 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.803894 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.803923 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.803999 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.804071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":45,"spinlock_wait_cycles":2944,"thread_start_us":90,"threads_started":1}
I20260925 16:40:54.804335 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.800324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:41457 (local address 127.0.97.193:36361)
0925 16:40:54.800609 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.800613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.800627 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:40:54.800707 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.800710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.800764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.800858 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:40:54.801126 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.801768 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803173 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.803825 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803932 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.804084 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.804124 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.804200 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
I20260925 16:40:54.804279 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.800133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.97.194:41457)
0925 16:40:54.800475 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.800488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:54.800589 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:54.800878 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:54.800881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:54.800898 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:40:54.801019 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.801025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.801892 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.801895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.803055 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:54.803065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.803934 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.803937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:54.804042 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.804060 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:40:54.804113 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 16:40:54.804166 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":20,"spinlock_wait_cycles":21632,"thread_start_us":72,"threads_started":1}
I20260925 16:40:54.804774 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979786bf10ad4c7f8521761a5c14093a" is_pre_election: true
I20260925 16:40:54.804950 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" is_pre_election: true
I20260925 16:40:54.805050 15084 raft_consensus.cc:2454] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 0.
I20260925 16:40:54.805184 14814 raft_consensus.cc:2454] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 0.
I20260925 16:40:54.805428 14888 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a; no voters:
I20260925 16:40:54.805693 15149 raft_consensus.cc:2781] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:40:54.805780 15149 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:40:54.805826 15149 raft_consensus.cc:3037] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:54.807159 15149 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.807482 15149 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 election: Requested vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361)
I20260925 16:40:54.807821 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979786bf10ad4c7f8521761a5c14093a"
I20260925 16:40:54.807855 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7117563ba79e44a0b381b9a154a3e1f5" candidate_uuid: "93636cbb558d4dbf8b8085f66d906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
I20260925 16:40:54.807943 15084 raft_consensus.cc:3037] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:54.807987 14814 raft_consensus.cc:3037] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:40:54.809145 15084 raft_consensus.cc:2454] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 1.
I20260925 16:40:54.809145 14814 raft_consensus.cc:2454] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93636cbb558d4dbf8b8085f66d906458 in term 1.
I20260925 16:40:54.809410 14888 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28, 93636cbb558d4dbf8b8085f66d906458; no voters:
I20260925 16:40:54.809561 15149 raft_consensus.cc:2781] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:40:54.809757 15149 raft_consensus.cc:686] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 1 LEADER]: Becoming Leader. State: Replica: 93636cbb558d4dbf8b8085f66d906458, State: Running, Role: LEADER
I20260925 16:40:54.810094 15149 consensus_queue.cc:237] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:54.813205 14670 catalog_manager.cc:5495] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "93636cbb558d4dbf8b8085f66d906458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } health_report { overall_health: UNKNOWN } } }
I20260925 16:40:54.924647 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.921157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.97.194:35547)
0925 16:40:54.921434 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:54.921438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:54.921452 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:40:54.921525 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:54.921527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:54.921578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:54.921676 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:40:54.921885 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.922385 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.923263 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:54.923471 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:54.923542 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:54.923799 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:40:54.924198 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:40:54.924295 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:54.924398 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 16:40:54.924445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260925 16:40:54.930387 15084 raft_consensus.cc:1261] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Refusing update from remote peer 93636cbb558d4dbf8b8085f66d906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:54.930420 14814 raft_consensus.cc:1261] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Refusing update from remote peer 93636cbb558d4dbf8b8085f66d906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:40:54.931056 15149 consensus_queue.cc:1035] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:54.931239 15153 consensus_queue.cc:1035] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:40:54.938087 15160 mvcc.cc:205] Tried to move back new op lower bound from 7333291847387234304 to 7333291846904426496. Current Snapshot: MvccSnapshot[applied={T|T < 7333291847387234304}]
I20260925 16:40:54.938525 15157 mvcc.cc:205] Tried to move back new op lower bound from 7333291847387234304 to 7333291846904426496. Current Snapshot: MvccSnapshot[applied={T|T < 7333291847387234304}]
I20260925 16:40:54.946981 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:54.947153 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:40:54.949442 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Sending DeleteTablet for 3 replicas of tablet 7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:40:54.950207 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:54 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42164
I20260925 16:40:54.950268 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:54 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:49668
I20260925 16:40:54.950759 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14710
I20260925 16:40:54.951457 15170 tablet_replica.cc:330] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: stopping tablet replica
I20260925 16:40:54.951676 14932 tablet_service.cc:1514] Processing DeleteTablet for tablet 7117563ba79e44a0b381b9a154a3e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:40:54 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:51810
I20260925 16:40:54.951788 15170 raft_consensus.cc:2229] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:40:54.952105 15170 raft_consensus.cc:2258] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:54.952119 15171 tablet_replica.cc:330] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: stopping tablet replica
I20260925 16:40:54.952509 15171 raft_consensus.cc:2229] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:40:54.952930 15171 raft_consensus.cc:2258] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:40:54.954293 15170 ts_tablet_manager.cc:1905] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:54.954293 15171 ts_tablet_manager.cc:1905] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:40:54.957078 15170 ts_tablet_manager.cc:1918] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:54.957180 15170 log.cc:1196] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:40:54.957566 15170 ts_tablet_manager.cc:1939] T 7117563ba79e44a0b381b9a154a3e1f5 P 979786bf10ad4c7f8521761a5c14093a: Deleting consensus metadata
I20260925 16:40:54.958040 15171 ts_tablet_manager.cc:1918] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:40:54.958200 15171 log.cc:1196] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:40:54.958629 15171 ts_tablet_manager.cc:1939] T 7117563ba79e44a0b381b9a154a3e1f5 P 93636cbb558d4dbf8b8085f66d906458: Deleting consensus metadata
I20260925 16:40:54.958843 14659 catalog_manager.cc:4841] TS 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): tablet 7117563ba79e44a0b381b9a154a3e1f5 (table test-table [id=b24ebfa07949434c8ff6e5338ef8960e]) successfully deleted
I20260925 16:40:54.959885 14659 catalog_manager.cc:4841] TS 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): tablet 7117563ba79e44a0b381b9a154a3e1f5 (table test-table [id=b24ebfa07949434c8ff6e5338ef8960e]) successfully deleted
W20260925 16:40:54.960247 14659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:36361 (error 108)
I20260925 16:40:54.960564 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14864
I20260925 16:40:54.962745 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:54.962330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36361 (local address 127.0.0.1:42180)
0925 16:40:54.962578 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:54.962647 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
W20260925 16:40:54.962917 14659 catalog_manager.cc:4584] TS 4bb1d7785c0e4101b96b3e9ed9c39a28 (127.0.97.193:36361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7117563ba79e44a0b381b9a154a3e1f5: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36361: connect: Connection refused (error 111)
I20260925 16:40:54.965670 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15001
I20260925 16:40:54.970966 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 14639
I20260925 16:40:54.977145 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--webserver_interface=127.0.97.254
--webserver_port=39017
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:40:55.085806 15174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:55.086063 15174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:55.086138 15174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:55.090138 15174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:40:55.090205 15174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:55.090225 15174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:55.090271 15174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:40:55.090302 15174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:40:55.095606 15174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:55.096632 15174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:55.097682 15174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:55.106021 15180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.106021 15182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.106030 15179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:55.106330 15174 server_base.cc:1034] running on GCE node
I20260925 16:40:55.106894 15174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:55.107972 15174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:55.109160 15174 hybrid_clock.cc:648] HybridClock initialized: now 1790354455109138 us; error 36 us; skew 500 ppm
Sep 25 16:40:55 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:55.111784 15174 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:40:55.112706 15174 webserver.cc:460] Webserver started at http://127.0.97.254:39017/ using document root <none> and password file <none>
I20260925 16:40:55.113255 15174 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:55.113396 15174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:55.117000 15174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:40:55.119093 15189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.120349 15174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:55.120440 15174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "b0e1015b3e3b4f1382d2f45610a85190"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "d8560c80d9dac3577d39652dcc6dde8d"
server_key_iv: "94af37011c26d32fc1ee9164940845df"
server_key_version: "encryptionkey@0"
I20260925 16:40:55.120920 15174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:55.139763 15174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:55.142666 15174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:55.142841 15174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:55.149468 15174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36959
I20260925 16:40:55.149797 15241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36959 every 8 connection(s)
I20260925 16:40:55.150542 15174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:40:55.153199 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15174
I20260925 16:40:55.153641 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:36361
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=45921
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:55.154996 15242 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:40:55.157078 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap starting.
I20260925 16:40:55.174360 15242 log.cc:824] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:55.185158 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:40:55.185556 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190: Bootstrap complete.
I20260925 16:40:55.189553 15242 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:55.189922 15242 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Initialized, Role: FOLLOWER
I20260925 16:40:55.190434 15242 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:55.190547 15242 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:40:55.190629 15242 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:40:55.190745 15242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:40:55.192538 15242 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:55.192860 15242 leader_election.cc:302] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e1015b3e3b4f1382d2f45610a85190; no voters:
I20260925 16:40:55.193195 15242 leader_election.cc:288] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:40:55.193292 15246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:40:55.193481 15246 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [term 3 LEADER]: Becoming Leader. State: Replica: b0e1015b3e3b4f1382d2f45610a85190, State: Running, Role: LEADER
I20260925 16:40:55.193770 15246 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } }
I20260925 16:40:55.194416 15242 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:40:55.194662 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e1015b3e3b4f1382d2f45610a85190. Latest consensus state: current_term: 3 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:55.194778 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:55.195173 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b0e1015b3e3b4f1382d2f45610a85190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e1015b3e3b4f1382d2f45610a85190" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36959 } } }
I20260925 16:40:55.195389 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e1015b3e3b4f1382d2f45610a85190 [sys.catalog]: This master's current role is: LEADER
I20260925 16:40:55.196638 15250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:40:55.200659 15250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:40:55.201177 15250 catalog_manager.cc:1259] Loaded cluster ID: 973b85691ba3451ca8af6fda7cbd32bc
I20260925 16:40:55.201249 15250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 16:40:55.202047 15250 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fabb63f19ad google::LogMessage::Fail() at ??:0
@ 0x7fabb63f488a google::LogMessage::SendToLog() at ??:0
@ 0x7fabb63f14de google::LogMessage::Flush() at ??:0
@ 0x7fabb63f2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fabc0d30fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fabc0d31d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fabc0d2e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fabc0d72640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fabb86433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fabb8643c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fabb86453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fabb8634512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabb9c786db start_thread at ??:0
@ 0x7fabb4eef61f clone at ??:0
W20260925 16:40:55.309676 15244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:55.310097 15244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:55.310240 15244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:55.316828 15244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:55.316936 15244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:55.317035 15244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:40:55.325389 15244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:36361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=45921
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:55.326823 15244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:55.328207 15244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:55.339358 15283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.339378 15284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.339638 15286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:55.339617 15244 server_base.cc:1034] running on GCE node
I20260925 16:40:55.340143 15244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:55.340837 15244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:55.342016 15244 hybrid_clock.cc:648] HybridClock initialized: now 1790354455341990 us; error 46 us; skew 500 ppm
Sep 25 16:40:55 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:55.344735 15244 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:40:55.345628 15244 webserver.cc:460] Webserver started at http://127.0.97.193:45921/ using document root <none> and password file <none>
I20260925 16:40:55.346187 15244 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:55.346276 15244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:55.350301 15244 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:40:55.352875 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.354229 15244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:40:55.354364 15244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28"
format_stamp: "Formatted at 2026-09-25 16:40:52 on dist-test-slave-kxl8"
server_key: "ff986f27904e6481565198d8107b74ba"
server_key_iv: "78bbc3888ffa2350ccb9e9920bed0faa"
server_key_version: "encryptionkey@0"
I20260925 16:40:55.354915 15244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:55.373091 15244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:55.376319 15244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:55.376528 15244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:55.377122 15244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:55.378304 15300 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:40:55.383589 15244 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:40:55.383688 15244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 16:40:55.383796 15244 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:40:55.385509 15244 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:40:55.385574 15244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 16:40:55.385932 15300 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap starting.
I20260925 16:40:55.399173 15244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:36361
I20260925 16:40:55.399387 15407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:36361 every 8 connection(s)
I20260925 16:40:55.400199 15244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:40:55.401173 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15244
I20260925 16:40:55.401707 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:35547
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=41859
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:55.409252 15300 log.cc:824] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Log is configured to *not* fsync() on all Append() calls
I20260925 16:40:55.418426 15300 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:40:55.418841 15300 tablet_bootstrap.cc:492] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Bootstrap complete.
I20260925 16:40:55.420280 15300 ts_tablet_manager.cc:1397] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.020s
I20260925 16:40:55.423560 15300 raft_consensus.cc:348] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:55.424074 15300 raft_consensus.cc:729] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb1d7785c0e4101b96b3e9ed9c39a28, State: Initialized, Role: FOLLOWER
I20260925 16:40:55.424583 15300 consensus_queue.cc:260] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:55.425609 15300 ts_tablet_manager.cc:1428] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28: Time spent starting tablet: real 0.005s user 0.006s sys 0.002s
W20260925 16:40:55.474911 15411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:40:55.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:54955)
0925 16:40:55.404749 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.404806 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:55.405773 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:55.474084 (+ 68311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1238,"thread_start_us":306,"threads_started":1}
W20260925 16:40:55.475447 15408 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:36959 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:36959: Client connection negotiation failed: client connection to 127.0.97.254:36959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:40:55.476178 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.475924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:48701)
0925 16:40:55.476050 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.476125 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 16:40:55.476786 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.476574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:33993)
0925 16:40:55.476688 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.476732 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 16:40:55.476946 15408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:40:55.520272 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:55.520546 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:55.520649 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:55.525198 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:55.525309 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:55.525424 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:40:55.530820 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:35547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=41859
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:55.531894 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:55.532904 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:55.542251 15418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.542259 15421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.542258 15419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:55.542575 15410 server_base.cc:1034] running on GCE node
I20260925 16:40:55.543041 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:55.543670 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:55.544829 15410 hybrid_clock.cc:648] HybridClock initialized: now 1790354455544816 us; error 25 us; skew 500 ppm
Sep 25 16:40:55 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:55.547529 15410 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:40:55.548403 15410 webserver.cc:460] Webserver started at http://127.0.97.194:41859/ using document root <none> and password file <none>
I20260925 16:40:55.548966 15410 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:55.549057 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:55.552836 15410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:40:55.554816 15428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.555981 15410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:40:55.556111 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "93636cbb558d4dbf8b8085f66d906458"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "acb144e256636ffe182a169617e9e544"
server_key_iv: "a93b562ba8784df5e9ba978778019832"
server_key_version: "encryptionkey@0"
I20260925 16:40:55.556660 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:55.574333 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:55.577456 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:55.577680 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:55.578321 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:55.579250 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:55.579342 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.579442 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:55.579501 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.588881 15410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:35547
I20260925 16:40:55.588904 15541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:35547 every 8 connection(s)
I20260925 16:40:55.590015 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:40:55.590266 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15410
I20260925 16:40:55.590792 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:45815
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:36959
--webserver_port=36497
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:40:55.594285 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.592385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:50213)
0925 16:40:55.592907 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.593013 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
W20260925 16:40:55.594781 15542 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:36959 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:36959: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
I20260925 16:40:55.595448 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.595237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:55465)
0925 16:40:55.595349 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.595394 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 16:40:55.596073 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.595847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:58837)
0925 16:40:55.595951 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.596001 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 16:40:55.596238 15542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:40:55.712951 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:40:55.713251 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:40:55.713356 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:40:55.717434 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:40:55.717519 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:40:55.717620 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:40:55.722971 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:45815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=36497
--tserver_master_addrs=127.0.97.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:40:55.724061 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:40:55.725098 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:40:55.734572 15550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:55.734645 15544 server_base.cc:1034] running on GCE node
W20260925 16:40:55.734668 15551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:40:55.734588 15553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:40:55.735147 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:40:55.735816 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:40:55.737004 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790354455736977 us; error 36 us; skew 500 ppm
Sep 25 16:40:55 dist-test-slave-kxl8 krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:40:55.739784 15544 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:40:55.740571 15544 webserver.cc:460] Webserver started at http://127.0.97.195:36497/ using document root <none> and password file <none>
I20260925 16:40:55.741072 15544 fs_manager.cc:359] Metadata directory not provided
I20260925 16:40:55.741138 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:40:55.744808 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:40:55.746781 15560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.747948 15544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:40:55.748040 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "979786bf10ad4c7f8521761a5c14093a"
format_stamp: "Formatted at 2026-09-25 16:40:53 on dist-test-slave-kxl8"
server_key: "f5754fee42ec9c037e1fa597ddc0e17c"
server_key_iv: "39b4eb4bb421481bb7940cdd26e259de"
server_key_version: "encryptionkey@0"
I20260925 16:40:55.748540 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:40:55.781563 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:40:55.784868 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:40:55.785022 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:40:55.785579 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:40:55.786372 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:40:55.786432 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.786490 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:40:55.786515 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:40:55.795346 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:45815
I20260925 16:40:55.795373 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:45815 every 8 connection(s)
I20260925 16:40:55.796298 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:40:55.799373 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15544
I20260925 16:40:55.800498 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.798377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:57163)
0925 16:40:55.798819 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.798941 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
W20260925 16:40:55.800972 15674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:36959 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:36959: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
I20260925 16:40:55.801669 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.801464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:59739)
0925 16:40:55.801562 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.801606 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 16:40:55.802251 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:55.802046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:46397)
0925 16:40:55.802148 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:55.802196 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:40:55.802413 15674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:40:56.478070 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.477623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:49691)
0925 16:40:56.477895 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:56.477971 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260925 16:40:56.597227 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.596838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:50135)
0925 16:40:56.597077 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:56.597135 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260925 16:40:56.791724 15680 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:56.791957 15680 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:56.793023 15680 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
Sep 25 16:40:56 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.195@KRBTEST.COM
I20260925 16:40:56.803320 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.802905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:58779)
0925 16:40:56.803154 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:56.803210 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":58,"threads_started":1}
Sep 25 16:40:56 dist-test-slave-kxl8 krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.194@KRBTEST.COM
I20260925 16:40:56.809435 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.793263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:45815 (local address 127.0.97.193:56165)
0925 16:40:56.793398 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:56.793416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:56.793527 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:56.795664 (+ 2137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:56.795671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:56.796285 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:56.796779 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:56.796792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.798818 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.798822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:56.799868 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:56.799879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.800836 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:56.801098 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:56.801842 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:56.801864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:56.803874 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:56.806491 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:56.806498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:56.806512 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:56.806971 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:56.807376 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:56.807381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:56.807599 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:56.808347 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:56.808354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:56.808718 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 16:40:56.808928 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 16:40:56.809235 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":194,"spinlock_wait_cycles":77056}
I20260925 16:40:56.809641 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.793460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:56165 (local address 127.0.97.195:45815)
0925 16:40:56.793808 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:56.793823 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:56.793862 (+ 39us) server_negotiation.cc:407] Connection header received
0925 16:40:56.794760 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:56.794769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:56.795264 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:56.795581 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:56.796960 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.798669 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800016 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.800706 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800873 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:56.804167 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:56.804172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:56.804184 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:56.804210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:56.806344 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807143 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:56.807152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:56.807159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:56.807290 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807700 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:56.807703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:56.807705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:56.808241 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:56.808353 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:56.809176 (+ 823us) server_negotiation.cc:299] Negotiation successful
0925 16:40:56.809474 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260925 16:40:56.810109 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.793775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:53113 (local address 127.0.97.194:35547)
0925 16:40:56.794110 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 16:40:56.794130 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:40:56.794162 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:40:56.794923 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:40:56.794929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:40:56.795253 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:40:56.795580 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:40:56.796947 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.798246 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800011 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.800706 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800874 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:56.804459 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:40:56.804465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:40:56.804481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:40:56.804521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:40:56.806371 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807085 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:56.807091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:56.807098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:56.807252 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807700 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:40:56.807703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:40:56.807705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:40:56.808210 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:40:56.808353 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:40:56.809715 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0925 16:40:56.809986 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":130,"threads_started":1}
I20260925 16:40:56.810496 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:56.793567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:35547 (local address 127.0.97.193:53113)
0925 16:40:56.793863 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:56.793879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:40:56.793984 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:40:56.796052 (+ 2068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:40:56.796055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:40:56.796282 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:40:56.796775 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:56.796790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.798414 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.798418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:56.799868 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:40:56.799879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:40:56.800836 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:40:56.800840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:40:56.801104 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:40:56.801856 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:40:56.801868 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:40:56.804273 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:40:56.806491 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:56.806498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:56.806512 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:56.806965 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:56.807371 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:40:56.807374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:40:56.807375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:40:56.807599 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:40:56.808347 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:40:56.808354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:40:56.808718 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 16:40:56.808928 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 16:40:56.810379 (+ 1451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":210,"spinlock_wait_cycles":15232,"thread_start_us":73,"threads_started":1}
W20260925 16:40:56.812134 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
W20260925 16:40:56.812383 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:40:56.812481 15297 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28; no voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a
I20260925 16:40:56.812709 15680 raft_consensus.cc:2726] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:40:57.479348 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:57.478905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:56963)
0925 16:40:57.479182 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:57.479243 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260925 16:40:57.598340 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:57.597956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:38147)
0925 16:40:57.598193 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:57.598250 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260925 16:40:57.804445 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:57.804026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:56985)
0925 16:40:57.804282 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:57.804340 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260925 16:40:58.480588 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:58.480179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:40343)
0925 16:40:58.480431 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:58.480493 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260925 16:40:58.599613 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:58.599129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:38175)
0925 16:40:58.599435 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:58.599502 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":74,"threads_started":1}
I20260925 16:40:58.790405 15690 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:40:58.790525 15690 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:40:58.790910 15690 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
W20260925 16:40:58.791775 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
W20260925 16:40:58.791958 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:40:58.792027 15297 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28; no voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a
I20260925 16:40:58.792191 15690 raft_consensus.cc:2726] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:40:58.805636 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:58.805208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:35159)
0925 16:40:58.805492 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:58.805545 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":125,"threads_started":1}
I20260925 16:40:59.482018 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:59.481607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:46309)
0925 16:40:59.481862 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:59.481923 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260925 16:40:59.600728 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:59.600300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:54345)
0925 16:40:59.600580 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:59.600638 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260925 16:40:59.806905 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:40:59.806506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:45817)
0925 16:40:59.806752 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:40:59.806810 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260925 16:41:00.483434 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:00.482972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:59501)
0925 16:41:00.483279 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:00.483340 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260925 16:41:00.602033 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:00.601630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:40295)
0925 16:41:00.601883 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:00.601940 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260925 16:41:00.808135 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:00.807723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:47063)
0925 16:41:00.807988 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:00.808042 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260925 16:41:01.484639 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:01.484213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:56883)
0925 16:41:01.484479 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:01.484534 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260925 16:41:01.603153 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:01.602718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:56315)
0925 16:41:01.602957 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:01.603008 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260925 16:41:01.753455 15700 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:01.753573 15700 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:41:01.753988 15700 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
W20260925 16:41:01.754771 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
W20260925 16:41:01.754935 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:41:01.754994 15297 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28; no voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a
I20260925 16:41:01.755184 15700 raft_consensus.cc:2726] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:41:01.809353 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:01.808889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:47917)
0925 16:41:01.809131 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:01.809185 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260925 16:41:02.486035 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:02.485597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:60969)
0925 16:41:02.485875 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:02.485935 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260925 16:41:02.604482 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:02.604016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:33503)
0925 16:41:02.604332 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:02.604388 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":88,"threads_started":1}
I20260925 16:41:02.810518 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:02.810052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:33779)
0925 16:41:02.810369 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:02.810427 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260925 16:41:03.487363 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:03.486946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:54783)
0925 16:41:03.487209 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:03.487268 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260925 16:41:03.605783 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:03.605393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:40049)
0925 16:41:03.605631 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:03.605690 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260925 16:41:03.811647 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:03.811274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:35549)
0925 16:41:03.811505 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:03.811558 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260925 16:41:04.488534 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:04.488097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:52471)
0925 16:41:04.488377 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:04.488435 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260925 16:41:04.606938 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:04.606491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:43425)
0925 16:41:04.606780 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:04.606837 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260925 16:41:04.812855 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:04.812473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:33727)
0925 16:41:04.812713 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:04.812764 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260925 16:41:05.489884 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:05.489482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:54203)
0925 16:41:05.489730 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:05.489789 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260925 16:41:05.608127 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:05.607698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:58213)
0925 16:41:05.607982 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:05.608034 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260925 16:41:05.814054 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:05.813587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:34965)
0925 16:41:05.813909 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:05.813962 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":84,"threads_started":1}
I20260925 16:41:05.957542 15714 raft_consensus.cc:482] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:05.957674 15714 raft_consensus.cc:504] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979786bf10ad4c7f8521761a5c14093a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 45815 } } peers { permanent_uuid: "93636cbb558d4dbf8b8085f66d906458" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 35547 } } peers { permanent_uuid: "4bb1d7785c0e4101b96b3e9ed9c39a28" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36361 } }
I20260925 16:41:05.958101 15714 leader_election.cc:288] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815), 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547)
W20260925 16:41:05.958937 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979786bf10ad4c7f8521761a5c14093a (127.0.97.195:45815): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
W20260925 16:41:05.959048 15297 leader_election.cc:341] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93636cbb558d4dbf8b8085f66d906458 (127.0.97.194:35547): Not found: Tablet not found: 7117563ba79e44a0b381b9a154a3e1f5
I20260925 16:41:05.959086 15297 leader_election.cc:302] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb1d7785c0e4101b96b3e9ed9c39a28; no voters: 93636cbb558d4dbf8b8085f66d906458, 979786bf10ad4c7f8521761a5c14093a
I20260925 16:41:05.959280 15714 raft_consensus.cc:2726] T 7117563ba79e44a0b381b9a154a3e1f5 P 4bb1d7785c0e4101b96b3e9ed9c39a28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:41:06.490994 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:06.490606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:48729)
0925 16:41:06.490844 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:06.490899 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260925 16:41:06.609341 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:06.608849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:48557)
0925 16:41:06.609118 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:06.609175 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260925 16:41:06.815091 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:06.814718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:53479)
0925 16:41:06.814948 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:06.814997 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260925 16:41:07.492619 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:07.492179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:48653)
0925 16:41:07.492458 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:07.492522 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260925 16:41:07.610661 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:07.610218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:52739)
0925 16:41:07.610497 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:07.610564 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260925 16:41:07.816337 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:07.815914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:39541)
0925 16:41:07.816178 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:07.816242 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260925 16:41:08.493974 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:08.493559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:38077)
0925 16:41:08.493813 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:08.493871 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260925 16:41:08.611917 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:08.611516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:51063)
0925 16:41:08.611768 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:08.611823 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260925 16:41:08.817564 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:08.817100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:48625)
0925 16:41:08.817418 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:08.817471 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":77,"threads_started":1}
I20260925 16:41:09.495257 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:09.494796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:39919)
0925 16:41:09.495078 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:09.495136 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260925 16:41:09.613027 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:09.612640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:45563)
0925 16:41:09.612880 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:09.612932 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260925 16:41:09.818821 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:09.818400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.195:49215)
0925 16:41:09.818673 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:09.818728 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260925 16:41:10.496707 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:10.496298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.193:55097)
0925 16:41:10.496548 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:10.496605 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260925 16:41:10.614565 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:10.614073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36959 (local address 127.0.97.194:34065)
0925 16:41:10.614404 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:10.614466 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":142,"threads_started":1}
I20260925 16:41:10.805454 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15244
I20260925 16:41:10.811375 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15410
I20260925 16:41:10.816501 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15544
2026-09-25T16:41:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19541 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:41:10 dist-test-slave-kxl8 krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:41:10 dist-test-slave-kxl8 krb5kdc[15731](info): set up 2 sockets
Sep 25 16:41:10 dist-test-slave-kxl8 krb5kdc[15731](info): commencing operation
krb5kdc: starting...
W20260925 16:41:11.858059 391 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:41:11 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:41:11.980937 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:12.089293 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:12.089614 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:12.089689 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:12.093748 15744 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:41:12.093832 15744 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:41:12.093858 15744 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:41:12.093880 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:41:12.093917 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:12.093962 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:12.094000 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:41:12.094038 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:41:12.099327 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:12.100415 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:12.101478 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:12.110491 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.110639 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.110561 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:12.110543 15744 server_base.cc:1034] running on GCE node
I20260925 16:41:12.111325 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:12.112422 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:12.113598 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790354472113561 us; error 49 us; skew 500 ppm
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:12.116608 15744 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:41:12.117589 15744 webserver.cc:460] Webserver started at http://127.0.97.254:33909/ using document root <none> and password file <none>
I20260925 16:41:12.118156 15744 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:12.118243 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:12.118470 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:41:12.120522 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "debca3c9b52e47438b44ef04cd1eb2b5"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "b0735fa6dab5eab32b958f7c74703a69"
server_key_iv: "04214a71bf3eae1365e197abe9712577"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.121083 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "debca3c9b52e47438b44ef04cd1eb2b5"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "b0735fa6dab5eab32b958f7c74703a69"
server_key_iv: "04214a71bf3eae1365e197abe9712577"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.124728 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:41:12.127156 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.128337 15744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:12.128476 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "debca3c9b52e47438b44ef04cd1eb2b5"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "b0735fa6dab5eab32b958f7c74703a69"
server_key_iv: "04214a71bf3eae1365e197abe9712577"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.128585 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:12.145413 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:12.148713 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:12.148923 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:12.155953 15744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:34289
I20260925 16:41:12.156244 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:34289 every 8 connection(s)
I20260925 16:41:12.156931 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:41:12.159896 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:12.165134 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap starting.
I20260925 16:41:12.166639 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15744
I20260925 16:41:12.166775 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:41:12.167061 391 external_mini_cluster.cc:1428] Setting key 9a59758cf09fc09901bfa5565e5a1043
I20260925 16:41:12.167503 15812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:12.168351 15812 log.cc:824] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:12.170214 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: No bootstrap required, opened a new log
I20260925 16:41:12.173524 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:12.173758 15812 raft_consensus.cc:374] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:12.173844 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Initialized, Role: FOLLOWER
I20260925 16:41:12.174290 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:12.174431 15812 raft_consensus.cc:388] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:41:12.174515 15812 raft_consensus.cc:482] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:41:12.174614 15812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:12.175575 15812 raft_consensus.cc:504] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:12.175948 15812 leader_election.cc:302] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debca3c9b52e47438b44ef04cd1eb2b5; no voters:
I20260925 16:41:12.176288 15812 leader_election.cc:288] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:41:12.176374 15817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:41:12.176553 15817 raft_consensus.cc:686] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 LEADER]: Becoming Leader. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Running, Role: LEADER
I20260925 16:41:12.176898 15817 consensus_queue.cc:237] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:12.177570 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:41:12.178488 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debca3c9b52e47438b44ef04cd1eb2b5. Latest consensus state: current_term: 1 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:12.178617 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:12.179018 15826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:41:12.178979 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:12.179076 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:12.182358 15826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:41:12.182544 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.168165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.97.254:34289)
0925 16:41:12.168716 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.168738 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.168784 (+ 46us) server_negotiation.cc:407] Connection header received
0925 16:41:12.169566 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.169586 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.169902 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.170234 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:12.171007 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.171996 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.172875 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.173403 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.173568 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.176517 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:12.176544 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:12.176566 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:12.176601 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:12.178727 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.179347 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.179353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.179358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.179493 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.179837 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.179840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.179842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.180331 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:12.181102 (+ 771us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.181266 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.181614 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":129,"threads_started":1}
I20260925 16:41:12.187893 15826 catalog_manager.cc:1347] Generated new cluster ID: 3fa5f15e1840446c83e38e8586af5925
I20260925 16:41:12.187956 15826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:41:12.197772 15826 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:41:12.198786 15826 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:41:12.205734 15826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Generated new TSK 0
I20260925 16:41:12.206439 15826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:41:12.243073 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:12.367267 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:12.367523 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:12.367568 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:12.371675 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:12.371732 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:12.371825 15838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:41:12.377027 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:12.378124 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:12.379235 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:12.388613 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:12.388695 15838 server_base.cc:1034] running on GCE node
W20260925 16:41:12.388617 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.388929 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:12.389376 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:12.389995 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:12.391149 15838 hybrid_clock.cc:648] HybridClock initialized: now 1790354472391121 us; error 49 us; skew 500 ppm
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:12.393976 15838 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:41:12.394829 15838 webserver.cc:460] Webserver started at http://127.0.97.193:43981/ using document root <none> and password file <none>
I20260925 16:41:12.395399 15838 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:12.395489 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:12.395710 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:41:12.397804 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "10d4f49936d84059bf3e997b3a482fca"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "5e76d5b30e2c5da1f9bf75038965dd23"
server_key_iv: "6f9b9a956797aa0981d1927176cbb7cf"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.398370 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "10d4f49936d84059bf3e997b3a482fca"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "5e76d5b30e2c5da1f9bf75038965dd23"
server_key_iv: "6f9b9a956797aa0981d1927176cbb7cf"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.402138 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:41:12.404605 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.406011 15838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:12.406150 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "10d4f49936d84059bf3e997b3a482fca"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "5e76d5b30e2c5da1f9bf75038965dd23"
server_key_iv: "6f9b9a956797aa0981d1927176cbb7cf"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.406275 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:12.426108 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:12.429425 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:12.429625 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:12.430233 15838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:12.431046 15838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:12.431123 15838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.431195 15838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:12.431234 15838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.439826 15838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:40329
I20260925 16:41:12.439848 15966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:40329 every 8 connection(s)
I20260925 16:41:12.440744 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:41:12.450333 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15838
I20260925 16:41:12.450441 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:41:12.450719 391 external_mini_cluster.cc:1428] Setting key 745cff992406778bd3955f29a34ff709
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:12.455615 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.442524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:33931 (local address 127.0.97.254:34289)
0925 16:41:12.442651 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.442655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.443474 (+ 819us) server_negotiation.cc:407] Connection header received
0925 16:41:12.444826 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.444830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.444888 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.444991 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:12.446194 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.446833 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.448782 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.449008 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.449092 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.452066 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:12.452070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:12.452072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:12.452104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:12.453628 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.454135 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.454140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.454142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.454225 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.454553 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.454558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.454560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.454737 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:12.454838 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.455323 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.455449 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:41:12.456480 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.442891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:33931)
0925 16:41:12.443330 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.443372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.444595 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.445168 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.445174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.445583 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:12.446051 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.446065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.447003 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.447007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.448612 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.448625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.449168 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.449172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.449378 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.449959 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:12.449975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:12.451829 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:12.453758 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.453764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.453779 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.454032 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.454315 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.454319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.454322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.454462 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.454838 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:12.454844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:12.455090 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.455271 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.455593 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260925 16:41:12.457751 15967 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:12.457994 15967 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:12.458483 15967 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:41:12.460165 15776 ts_manager.cc:194] Registered new tserver with Master: 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329)
I20260925 16:41:12.461386 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:33931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:41:12.478353 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:12.590193 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:12.590502 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:12.590590 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:12.594733 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:12.594815 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:12.594901 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:41:12.600137 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:12.601189 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:12.602246 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:12.611761 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.611797 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.611768 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:12.611984 15972 server_base.cc:1034] running on GCE node
I20260925 16:41:12.612418 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:12.613041 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:12.614205 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790354472614174 us; error 47 us; skew 500 ppm
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:12.616842 15972 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:41:12.617736 15972 webserver.cc:460] Webserver started at http://127.0.97.194:34385/ using document root <none> and password file <none>
I20260925 16:41:12.618278 15972 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:12.618364 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:12.618579 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:41:12.620500 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "db8bd5f05e76481681da6f16d6aab269"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "d1dc35243504434601812e070d66a1db"
server_key_iv: "6db0edc55c79a014017d7abcaa4a1396"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.621076 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "db8bd5f05e76481681da6f16d6aab269"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "d1dc35243504434601812e070d66a1db"
server_key_iv: "6db0edc55c79a014017d7abcaa4a1396"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.624622 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:41:12.626958 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.628171 15972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:12.628261 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "db8bd5f05e76481681da6f16d6aab269"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "d1dc35243504434601812e070d66a1db"
server_key_iv: "6db0edc55c79a014017d7abcaa4a1396"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.628343 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:12.640110 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:12.643193 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:12.643342 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:12.643869 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:12.644625 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:12.644678 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.644716 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:12.644739 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.653601 15972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:39641
I20260925 16:41:12.653622 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:39641 every 8 connection(s)
I20260925 16:41:12.654538 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:12.664340 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 15972
I20260925 16:41:12.664467 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:41:12.664736 391 external_mini_cluster.cc:1428] Setting key fbf61f0e1f2e696c2bab042d274c8bf1
I20260925 16:41:12.667958 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.656295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:51001 (local address 127.0.97.254:34289)
0925 16:41:12.656450 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.656454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.657321 (+ 867us) server_negotiation.cc:407] Connection header received
0925 16:41:12.658570 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.658573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.658629 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.658710 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:12.659820 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.660425 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.661576 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.661808 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.661889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.664219 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:12.664224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:12.664226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:12.664250 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:12.665848 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.666416 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.666420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.666423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.666518 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.666826 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.666833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.666837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.667093 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:12.667192 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.667661 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.667812 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:41:12.668992 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.656724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:51001)
0925 16:41:12.657142 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.657190 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.658381 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.658874 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.658880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.659225 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:12.659692 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.659705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.660535 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.660537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.661469 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.661479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.661932 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.661934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.662067 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.662642 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:12.662658 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:12.664042 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:12.665986 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.665992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.666005 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.666316 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.666606 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.666608 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.666610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.666737 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.667195 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:12.667224 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:12.667434 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.667581 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.667946 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260925 16:41:12.670225 16101 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:12.670477 16101 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:12.670954 16101 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
I20260925 16:41:12.672024 15776 ts_manager.cc:194] Registered new tserver with Master: db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
I20260925 16:41:12.672825 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:51001
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:41:12.692889 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:12.804270 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:12.804553 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:12.804597 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:12.808744 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:12.808804 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:12.808904 16106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:41:12.814517 16106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:12.815706 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:12.816731 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:41:12.826578 16106 server_base.cc:1034] running on GCE node
W20260925 16:41:12.826632 16112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.826610 16111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:12.826589 16114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:12.827298 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:12.827942 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:12.829154 16106 hybrid_clock.cc:648] HybridClock initialized: now 1790354472829119 us; error 48 us; skew 500 ppm
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:12.832685 16106 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:41:12.833599 16106 webserver.cc:460] Webserver started at http://127.0.97.195:46181/ using document root <none> and password file <none>
I20260925 16:41:12.834189 16106 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:12.834255 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:12.834438 16106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:41:12.836472 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "f7af947022f04ea0acd871dba14be977"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "ab0b6a0d0abbf3058adaa23da2e3e5d4"
server_key_iv: "d4d85fc8ad7bc0a931d8fb7f38f2b67f"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.836993 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "f7af947022f04ea0acd871dba14be977"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "ab0b6a0d0abbf3058adaa23da2e3e5d4"
server_key_iv: "d4d85fc8ad7bc0a931d8fb7f38f2b67f"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.840649 16106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:41:12.842995 16121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.844161 16106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:41:12.844307 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "f7af947022f04ea0acd871dba14be977"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "ab0b6a0d0abbf3058adaa23da2e3e5d4"
server_key_iv: "d4d85fc8ad7bc0a931d8fb7f38f2b67f"
server_key_version: "encryptionkey@0"
I20260925 16:41:12.844412 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:12.857072 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:12.860350 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:12.860566 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:12.861109 16106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:12.861914 16106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:12.861963 16106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.862006 16106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:12.862030 16106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:12.870710 16106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:34979
I20260925 16:41:12.870733 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:34979 every 8 connection(s)
I20260925 16:41:12.871615 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:41:12.879226 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16106
I20260925 16:41:12.879366 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:41:12.879635 391 external_mini_cluster.cc:1428] Setting key 812140272091d92fa0f0881788c9cffe
Sep 25 16:41:12 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:12.885010 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.873363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:51219 (local address 127.0.97.254:34289)
0925 16:41:12.873524 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.873529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.874807 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 16:41:12.875696 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.875699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.875747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.875828 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:12.876949 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.877594 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.878829 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.879049 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.879188 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.882056 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:12.882059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:12.882061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:12.882083 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:12.883268 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.883732 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.883734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.883736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.883818 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.884145 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.884147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.884148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.884309 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:12.884383 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.884803 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.884895 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:41:12.885908 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.873789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:51219)
0925 16:41:12.874653 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.874695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.875447 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.875984 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.875990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.876340 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:12.876803 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.876815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.877728 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.877731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.878716 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.878726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.879226 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.879231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.879462 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.880068 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:12.880104 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:12.881921 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:12.883379 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.883383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.883393 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.883640 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.883922 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:12.883925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:12.883926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:12.884064 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:12.884393 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:12.884398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:12.884604 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.884754 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.884986 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":558,"threads_started":1}
I20260925 16:41:12.886971 16235 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:12.887176 16235 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:12.887600 16235 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
I20260925 16:41:12.888535 15776 ts_manager.cc:194] Registered new tserver with Master: f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979)
I20260925 16:41:12.889110 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:51219
I20260925 16:41:12.893460 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:41:12.902266 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.895336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.0.97.254:34289)
0925 16:41:12.895448 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.895451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.895598 (+ 147us) server_negotiation.cc:407] Connection header received
0925 16:41:12.895695 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.895697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.895733 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.895808 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:12.896437 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.896934 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.897812 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.898029 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.898106 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.899388 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:12.899391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:12.899393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:12.899414 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:12.900772 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.901180 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.901182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.901183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.901257 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:12.901564 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:12.901566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:12.901567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:12.901777 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:12.901878 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.902066 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.902152 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:41:12.904915 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:41:12.907925 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:41:12.921450 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.915867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.97.195:34979)
0925 16:41:12.916322 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.916334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.916358 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:41:12.916459 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.916464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.916666 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.917174 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:12.917357 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.918493 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.919885 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.920743 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.920865 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.921017 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.921178 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.921269 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":146,"threads_started":1}
I20260925 16:41:12.921438 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.915733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.0.1:56992)
0925 16:41:12.916120 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.916173 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.916316 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.916844 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.916851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.916873 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:12.917014 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.917021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.918666 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.918669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.919769 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.919778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.920864 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.920868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.920985 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.921021 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.921148 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.921303 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":67,"threads_started":1}
I20260925 16:41:12.921897 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.916450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40329 (local address 127.0.0.1:32850)
0925 16:41:12.916686 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.916699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.916788 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.917642 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.917645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.917663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:12.917761 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.917766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.919089 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.919094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.920284 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.920294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.921521 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.921525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.921611 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.921648 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.921696 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.921761 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260925 16:41:12.922297 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.915922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.0.1:51274)
0925 16:41:12.916400 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.916419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.916505 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.917979 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.917988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.918004 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:12.918076 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.918081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.919355 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.919358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.920464 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.920472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.921898 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.921901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.921995 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.922024 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.922099 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.922168 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":21,"thread_start_us":106,"threads_started":1}
I20260925 16:41:12.922613 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.916807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.97.193:40329)
0925 16:41:12.917137 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.917152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.917174 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:41:12.917248 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.917253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.917486 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.917904 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:12.918054 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.918968 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.920419 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.921254 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.922163 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.922316 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.922372 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.922477 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260925 16:41:12.922820 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.916964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.0.97.194:39641)
0925 16:41:12.917320 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.917333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.917358 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:41:12.917437 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.917442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.917644 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.917797 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:12.918194 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.919218 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.920587 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.921596 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.921696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.921840 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.922551 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.922654 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":145,"threads_started":1}
I20260925 16:41:12.924891 15901 tablet_service.cc:1467] Processing CreateTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 (DEFAULT_TABLE table=test-table [id=46400e4e462e4d89b4ae79976dbb2445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:12.924906 16169 tablet_service.cc:1467] Processing CreateTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 (DEFAULT_TABLE table=test-table [id=46400e4e462e4d89b4ae79976dbb2445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:12.925221 16035 tablet_service.cc:1467] Processing CreateTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 (DEFAULT_TABLE table=test-table [id=46400e4e462e4d89b4ae79976dbb2445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:12.925833 15901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334731fb2a2b4da99b2eadde9c801de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:12.925935 16169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334731fb2a2b4da99b2eadde9c801de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:12.926090 16035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334731fb2a2b4da99b2eadde9c801de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:12.932646 16252 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap starting.
I20260925 16:41:12.933219 16253 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Bootstrap starting.
I20260925 16:41:12.935181 16252 tablet_bootstrap.cc:654] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:12.935181 16253 tablet_bootstrap.cc:654] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:12.935700 16254 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Bootstrap starting.
I20260925 16:41:12.936553 16252 log.cc:824] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:12.936553 16253 log.cc:824] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:12.938115 16254 tablet_bootstrap.cc:654] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:12.938655 16252 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: No bootstrap required, opened a new log
I20260925 16:41:12.938701 16253 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: No bootstrap required, opened a new log
I20260925 16:41:12.938858 16252 ts_tablet_manager.cc:1397] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:41:12.938898 16253 ts_tablet_manager.cc:1397] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:41:12.938966 16254 log.cc:824] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:12.940809 16254 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: No bootstrap required, opened a new log
I20260925 16:41:12.940982 16254 ts_tablet_manager.cc:1397] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:41:12.942021 16252 raft_consensus.cc:348] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.942250 16252 raft_consensus.cc:374] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:12.942301 16252 raft_consensus.cc:729] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d4f49936d84059bf3e997b3a482fca, State: Initialized, Role: FOLLOWER
I20260925 16:41:12.942721 16252 consensus_queue.cc:260] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.942790 16253 raft_consensus.cc:348] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.943046 16253 raft_consensus.cc:374] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:12.943123 16253 raft_consensus.cc:729] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7af947022f04ea0acd871dba14be977, State: Initialized, Role: FOLLOWER
I20260925 16:41:12.943450 15967 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
I20260925 16:41:12.943528 16253 consensus_queue.cc:260] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.943684 16252 ts_tablet_manager.cc:1428] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:41:12.944308 16235 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
I20260925 16:41:12.944456 16253 ts_tablet_manager.cc:1428] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 16:41:12.944797 16254 raft_consensus.cc:348] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.945051 16254 raft_consensus.cc:374] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:12.945103 16254 raft_consensus.cc:729] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8bd5f05e76481681da6f16d6aab269, State: Initialized, Role: FOLLOWER
I20260925 16:41:12.945550 16254 consensus_queue.cc:260] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.946323 16101 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
I20260925 16:41:12.946466 16254 ts_tablet_manager.cc:1428] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260925 16:41:12.989017 16258 raft_consensus.cc:482] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:12.989202 16258 raft_consensus.cc:504] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.990465 16258 leader_election.cc:288] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
I20260925 16:41:12.994807 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.990976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:37859 (local address 127.0.97.194:39641)
0925 16:41:12.991098 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.991101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.991211 (+ 110us) server_negotiation.cc:407] Connection header received
0925 16:41:12.991347 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.991349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.991391 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.991479 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:12.991776 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.992581 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.993792 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.994245 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.994369 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.994423 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.994588 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.994658 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 16:41:12.994836 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.990768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:42415 (local address 127.0.97.195:34979)
0925 16:41:12.990926 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:12.990931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:12.991025 (+ 94us) server_negotiation.cc:407] Connection header received
0925 16:41:12.991325 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:12.991330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:12.991385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:12.991458 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:12.991856 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.992580 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.993790 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.994268 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.994391 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.994444 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:12.994626 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 16:41:12.994713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 16:41:12.994869 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.990726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.97.193:37859)
0925 16:41:12.991113 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.991124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.991217 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.991503 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.991505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.991546 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:12.991651 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.991660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.992740 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.992742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.993648 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.993658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.994370 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.994373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.994473 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.994492 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.994574 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.994644 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":19,"spinlock_wait_cycles":23040,"thread_start_us":110,"threads_started":1}
I20260925 16:41:12.995065 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:12.990640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.97.193:42415)
0925 16:41:12.990929 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:12.990946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:12.991168 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:12.991483 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:12.991486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:12.991502 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:12.991685 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.991692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.992674 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.992677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.993671 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:12.993680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:12.994391 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:12.994395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:12.994504 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:12.994531 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:41:12.994610 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:41:12.994680 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":98,"threads_started":1}
I20260925 16:41:12.995532 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334731fb2a2b4da99b2eadde9c801de0" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8bd5f05e76481681da6f16d6aab269" is_pre_election: true
I20260925 16:41:12.995782 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334731fb2a2b4da99b2eadde9c801de0" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7af947022f04ea0acd871dba14be977" is_pre_election: true
I20260925 16:41:12.995877 16055 raft_consensus.cc:2454] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10d4f49936d84059bf3e997b3a482fca in term 0.
I20260925 16:41:12.996062 16189 raft_consensus.cc:2454] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10d4f49936d84059bf3e997b3a482fca in term 0.
I20260925 16:41:12.996372 15854 leader_election.cc:302] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca, db8bd5f05e76481681da6f16d6aab269; no voters:
I20260925 16:41:12.996651 16258 raft_consensus.cc:2781] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:41:12.996743 16258 raft_consensus.cc:482] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:41:12.996801 16258 raft_consensus.cc:3037] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:12.997805 16258 raft_consensus.cc:504] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:12.998116 16258 leader_election.cc:288] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 election: Requested vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
I20260925 16:41:12.998425 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334731fb2a2b4da99b2eadde9c801de0" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7af947022f04ea0acd871dba14be977"
I20260925 16:41:12.998435 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334731fb2a2b4da99b2eadde9c801de0" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8bd5f05e76481681da6f16d6aab269"
I20260925 16:41:12.998545 16189 raft_consensus.cc:3037] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:12.998545 16055 raft_consensus.cc:3037] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:12.999758 16189 raft_consensus.cc:2454] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d4f49936d84059bf3e997b3a482fca in term 1.
I20260925 16:41:12.999758 16055 raft_consensus.cc:2454] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d4f49936d84059bf3e997b3a482fca in term 1.
I20260925 16:41:13.000043 15857 leader_election.cc:302] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca, f7af947022f04ea0acd871dba14be977; no voters:
I20260925 16:41:13.000193 16258 raft_consensus.cc:2781] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:41:13.000420 16258 raft_consensus.cc:686] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 LEADER]: Becoming Leader. State: Replica: 10d4f49936d84059bf3e997b3a482fca, State: Running, Role: LEADER
I20260925 16:41:13.000763 16258 consensus_queue.cc:237] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:13.003935 15776 catalog_manager.cc:5495] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca reported cstate change: term changed from 0 to 1, leader changed from <none> to 10d4f49936d84059bf3e997b3a482fca (127.0.97.193). New cstate: current_term: 1 leader_uuid: "10d4f49936d84059bf3e997b3a482fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } health_report { overall_health: HEALTHY } } }
I20260925 16:41:13.018687 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:13.015453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.97.193:40329)
0925 16:41:13.015575 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:13.015583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:13.015599 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:41:13.015747 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:13.015749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:13.015800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:13.015884 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:41:13.016098 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:13.016582 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:13.017434 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:13.017638 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:13.017727 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:13.017958 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:41:13.018399 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:41:13.018488 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:13.018537 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 16:41:13.018583 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:41:13.024437 16189 raft_consensus.cc:1261] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Refusing update from remote peer 10d4f49936d84059bf3e997b3a482fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:41:13.024437 16055 raft_consensus.cc:1261] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Refusing update from remote peer 10d4f49936d84059bf3e997b3a482fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:41:13.025177 16258 consensus_queue.cc:1035] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:41:13.025405 16263 consensus_queue.cc:1035] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:41:13.041857 15774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:41:13.042052 15774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:41:13.044440 15774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Sending DeleteTablet for 3 replicas of tablet 334731fb2a2b4da99b2eadde9c801de0
I20260925 16:41:13.045221 16169 tablet_service.cc:1514] Processing DeleteTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:56992
I20260925 16:41:13.045245 15901 tablet_service.cc:1514] Processing DeleteTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:32850
I20260925 16:41:13.045508 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15838
I20260925 16:41:13.045729 16035 tablet_service.cc:1514] Processing DeleteTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:51274
I20260925 16:41:13.046489 16278 tablet_replica.cc:330] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: stopping tablet replica
I20260925 16:41:13.046762 16278 raft_consensus.cc:2229] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:41:13.047045 16278 raft_consensus.cc:2258] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:41:13.047416 16277 tablet_replica.cc:330] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: stopping tablet replica
I20260925 16:41:13.047667 16277 raft_consensus.cc:2229] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:41:13.047988 16277 raft_consensus.cc:2258] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:41:13.049053 16278 ts_tablet_manager.cc:1905] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:41:13.049378 16277 ts_tablet_manager.cc:1905] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:41:13.051537 16277 ts_tablet_manager.cc:1918] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:41:13.051640 16277 log.cc:1196] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/334731fb2a2b4da99b2eadde9c801de0
I20260925 16:41:13.051934 16277 ts_tablet_manager.cc:1939] T 334731fb2a2b4da99b2eadde9c801de0 P f7af947022f04ea0acd871dba14be977: Deleting consensus metadata
I20260925 16:41:13.052498 16278 ts_tablet_manager.cc:1918] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:41:13.052595 16278 log.cc:1196] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/334731fb2a2b4da99b2eadde9c801de0
I20260925 16:41:13.052932 16278 ts_tablet_manager.cc:1939] T 334731fb2a2b4da99b2eadde9c801de0 P db8bd5f05e76481681da6f16d6aab269: Deleting consensus metadata
I20260925 16:41:13.053054 15763 catalog_manager.cc:4841] TS f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): tablet 334731fb2a2b4da99b2eadde9c801de0 (table test-table [id=46400e4e462e4d89b4ae79976dbb2445]) successfully deleted
I20260925 16:41:13.053359 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15972
W20260925 16:41:13.053390 15760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:40329 (error 108)
I20260925 16:41:13.054767 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:13.054498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40329 (local address 127.0.0.1:32858)
0925 16:41:13.054619 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:13.054684 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:41:13.054932 15760 catalog_manager.cc:4584] TS 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 334731fb2a2b4da99b2eadde9c801de0: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40329: connect: Connection refused (error 111)
I20260925 16:41:13.058341 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16106
I20260925 16:41:13.058781 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:13.058468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.0.1:51278)
0925 16:41:13.058637 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:13.058689 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:39641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 16:41:13.063834 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 15744
I20260925 16:41:33.070115 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--webserver_interface=127.0.97.254
--webserver_port=33909
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:33.178817 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:33.179117 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:33.179203 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:33.183252 16280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:41:33.183336 16280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:41:33.183373 16280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:41:33.183408 16280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:41:33.183440 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:33.183485 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:33.183516 16280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:41:33.183545 16280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:41:33.188655 16280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:33.189721 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:33.190729 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:33.203584 16289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.203578 16286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.203584 16287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.203679 16280 server_base.cc:1034] running on GCE node
I20260925 16:41:33.204464 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:33.205780 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:33.206987 16280 hybrid_clock.cc:648] HybridClock initialized: now 1790354493206924 us; error 71 us; skew 500 ppm
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:33.215346 16280 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:41:33.216356 16280 webserver.cc:460] Webserver started at http://127.0.97.254:33909/ using document root <none> and password file <none>
I20260925 16:41:33.216971 16280 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:33.217082 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:33.221122 16280 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:41:33.223331 16296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.224575 16280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:33.224717 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "debca3c9b52e47438b44ef04cd1eb2b5"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "b0735fa6dab5eab32b958f7c74703a69"
server_key_iv: "04214a71bf3eae1365e197abe9712577"
server_key_version: "encryptionkey@0"
I20260925 16:41:33.225447 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:33.252918 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:33.256548 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:33.256753 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:33.264043 16280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:34289
I20260925 16:41:33.264343 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:34289 every 8 connection(s)
I20260925 16:41:33.265237 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:41:33.266499 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16280
I20260925 16:41:33.266932 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:40329
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=43981
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:41:33.269905 16349 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:41:33.271620 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap starting.
I20260925 16:41:33.288478 16349 log.cc:824] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:33.297345 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:41:33.297730 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap complete.
I20260925 16:41:33.301764 16349 raft_consensus.cc:348] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:33.302398 16349 raft_consensus.cc:729] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Initialized, Role: FOLLOWER
I20260925 16:41:33.302947 16349 consensus_queue.cc:260] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:33.303098 16349 raft_consensus.cc:388] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:41:33.303193 16349 raft_consensus.cc:482] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:41:33.303316 16349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:41:33.305186 16349 raft_consensus.cc:504] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:33.305547 16349 leader_election.cc:302] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debca3c9b52e47438b44ef04cd1eb2b5; no voters:
I20260925 16:41:33.305855 16349 leader_election.cc:288] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:41:33.305985 16353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:41:33.306218 16353 raft_consensus.cc:686] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 LEADER]: Becoming Leader. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Running, Role: LEADER
I20260925 16:41:33.306573 16353 consensus_queue.cc:237] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:33.307056 16349 sys_catalog.cc:564] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:41:33.307940 16354 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:33.308094 16354 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:33.308452 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debca3c9b52e47438b44ef04cd1eb2b5. Latest consensus state: current_term: 2 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:33.308599 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:33.308948 16362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:41:33.314013 16362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:41:33.314611 16362 catalog_manager.cc:1259] Loaded cluster ID: 3fa5f15e1840446c83e38e8586af5925
I20260925 16:41:33.314702 16362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:41:33.317096 16362 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:41:33.317782 16362 catalog_manager.cc:5879] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Loaded TSK: 0
I20260925 16:41:33.331845 16362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Generated new TSK 1
I20260925 16:41:33.332090 16362 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:41:33.384125 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:33.384377 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:33.384418 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:33.388453 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:33.388516 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:33.388576 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:41:33.393951 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=43981
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:33.394992 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:33.395969 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:33.406737 16379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.406733 16377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.407015 16351 server_base.cc:1034] running on GCE node
W20260925 16:41:33.406737 16376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.407651 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:33.408378 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:33.409570 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790354493409537 us; error 52 us; skew 500 ppm
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:33.412470 16351 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:41:33.413403 16351 webserver.cc:460] Webserver started at http://127.0.97.193:43981/ using document root <none> and password file <none>
I20260925 16:41:33.413987 16351 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:33.414098 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:33.418236 16351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:41:33.420579 16386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.421994 16351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:41:33.422133 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "10d4f49936d84059bf3e997b3a482fca"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "5e76d5b30e2c5da1f9bf75038965dd23"
server_key_iv: "6f9b9a956797aa0981d1927176cbb7cf"
server_key_version: "encryptionkey@0"
I20260925 16:41:33.422693 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:33.454054 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:33.457482 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:33.457698 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:33.458354 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:33.459576 16393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:41:33.463754 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:41:33.463840 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 16:41:33.463917 16351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:41:33.465585 16351 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:41:33.465658 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:33.466008 16393 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap starting.
I20260925 16:41:33.475982 16351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:40329
I20260925 16:41:33.476078 16500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:40329 every 8 connection(s)
I20260925 16:41:33.477056 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:41:33.483901 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16351
I20260925 16:41:33.484381 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:39641
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=34385
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:41:33.486027 16393 log.cc:824] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:33.494549 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.478983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:60379)
0925 16:41:33.479545 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.479602 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.480874 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.482140 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.482155 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.482670 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:33.483377 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.483405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.484950 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.484954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.486082 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.486097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.487170 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.487177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.487367 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.488063 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:33.488079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:33.489554 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:33.491684 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.491690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.491701 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.491957 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.492315 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.492317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.492319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.492454 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.493012 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:33.493018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:33.493258 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.493502 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.493810 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
I20260925 16:41:33.495807 16393 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:41:33.495913 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.479129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:60379 (local address 127.0.97.254:34289)
0925 16:41:33.479564 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.479583 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.479747 (+ 164us) server_negotiation.cc:407] Connection header received
0925 16:41:33.481376 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.481389 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.481686 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.481966 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.483595 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.484812 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.486242 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.487010 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.487197 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.489767 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.489772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.489785 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.489812 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.491545 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.492078 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.492088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.492092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.492208 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.492551 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.492555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.492557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.492895 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.493040 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.495002 (+ 1962us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.495313 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260925 16:41:33.496256 16393 tablet_bootstrap.cc:492] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap complete.
I20260925 16:41:33.497784 16393 ts_tablet_manager.cc:1397] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260925 16:41:33.497977 16501 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:33.498203 16501 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:33.498672 16501 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
I20260925 16:41:33.500867 16313 ts_manager.cc:194] Registered new tserver with Master: 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329)
I20260925 16:41:33.500656 16393 raft_consensus.cc:348] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.501168 16393 raft_consensus.cc:729] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d4f49936d84059bf3e997b3a482fca, State: Initialized, Role: FOLLOWER
I20260925 16:41:33.501664 16393 consensus_queue.cc:260] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.502728 16393 ts_tablet_manager.cc:1428] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:41:33.504594 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:60379
I20260925 16:41:33.507318 16501 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.193@KRBTEST.COM
I20260925 16:41:33.515573 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.503920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40329 (local address 127.0.0.1:60668)
0925 16:41:33.504376 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.504433 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.504565 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.505568 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.505574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.505915 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:33.506009 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.506016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.507087 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.507091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.507950 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.507960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.509510 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.509513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.509625 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.510175 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:33.510190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:33.511755 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:33.513874 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.513882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.513886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.514118 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.514430 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.514432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.514433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.514510 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.515060 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:33.515065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:33.515168 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.515287 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.515438 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260925 16:41:33.515720 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.504535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.97.193:40329)
0925 16:41:33.504946 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.504957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.504978 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:41:33.505041 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.505045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.505231 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.505405 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.506134 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.506975 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.508697 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.509263 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.509383 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.511889 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.511899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.511912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.511938 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.513749 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.514232 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.514238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.514240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.514334 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.514593 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.514596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.514598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.514895 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.515007 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.515384 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.515553 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":108,"threads_started":1}
I20260925 16:41:33.516693 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet 334731fb2a2b4da99b2eadde9c801de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:60668
I20260925 16:41:33.517572 16510 tablet_replica.cc:330] stopping tablet replica
I20260925 16:41:33.517866 16510 raft_consensus.cc:2229] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:41:33.518118 16510 raft_consensus.cc:2258] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:41:33.519052 16510 ts_tablet_manager.cc:1905] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:41:33.522624 16510 ts_tablet_manager.cc:1918] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:41:33.522703 16510 log.cc:1196] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal/wals/334731fb2a2b4da99b2eadde9c801de0
I20260925 16:41:33.522979 16510 ts_tablet_manager.cc:1939] T 334731fb2a2b4da99b2eadde9c801de0 P 10d4f49936d84059bf3e997b3a482fca: Deleting consensus metadata
I20260925 16:41:33.524971 16297 catalog_manager.cc:4841] TS 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329): tablet 334731fb2a2b4da99b2eadde9c801de0 (table test-table [id=46400e4e462e4d89b4ae79976dbb2445]) successfully deleted
W20260925 16:41:33.624585 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:33.624851 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:33.624893 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:33.629117 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:33.629179 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:33.629235 16505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:41:33.634413 16505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:39641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=34385
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:33.635349 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:33.636283 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:33.645632 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.645700 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.645741 16505 server_base.cc:1034] running on GCE node
W20260925 16:41:33.645897 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.646273 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:33.646927 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:33.648109 16505 hybrid_clock.cc:648] HybridClock initialized: now 1790354493648086 us; error 42 us; skew 500 ppm
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:33.650776 16505 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:41:33.651598 16505 webserver.cc:460] Webserver started at http://127.0.97.194:34385/ using document root <none> and password file <none>
I20260925 16:41:33.652151 16505 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:33.652256 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:33.655994 16505 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:41:33.658152 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.659349 16505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:33.659510 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "db8bd5f05e76481681da6f16d6aab269"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "d1dc35243504434601812e070d66a1db"
server_key_iv: "6db0edc55c79a014017d7abcaa4a1396"
server_key_version: "encryptionkey@0"
I20260925 16:41:33.660055 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:33.682574 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:33.685577 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:33.685778 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:33.686367 16505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:33.687175 16505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:33.687256 16505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.687323 16505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:33.687379 16505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.696691 16505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:39641
I20260925 16:41:33.696720 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:39641 every 8 connection(s)
I20260925 16:41:33.697722 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:41:33.702273 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16505
I20260925 16:41:33.702726 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:34979
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=46181
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:33.712538 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.699537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:47135 (local address 127.0.97.254:34289)
0925 16:41:33.699697 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.699701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.700464 (+ 763us) server_negotiation.cc:407] Connection header received
0925 16:41:33.701861 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.701864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.701928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.702023 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.703326 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.703866 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.704998 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.705245 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.705391 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.708460 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.708468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.708471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.708496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.710072 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.710781 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.710783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.710785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.710856 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.711329 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.711331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.711332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.711496 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.711567 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.712308 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.712414 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 16:41:33.714004 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.699882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:47135)
0925 16:41:33.700314 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.700356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.701636 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.702193 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.702200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.702611 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:33.703168 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.703182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.703997 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.704001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.704868 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.704878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.705429 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.705433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.705601 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.706209 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:33.706225 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:33.708237 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:33.710209 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.710219 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.710239 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.710664 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.710956 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.710962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.710966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.711213 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.711609 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:33.711618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:33.712002 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.712271 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.712685 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260925 16:41:33.715261 16639 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:33.715565 16639 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:33.716187 16639 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
I20260925 16:41:33.717409 16313 ts_manager.cc:194] Registered new tserver with Master: db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
I20260925 16:41:33.718006 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:47135
W20260925 16:41:33.822494 16642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:33.822831 16642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:33.822919 16642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:33.827188 16642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:33.827263 16642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:33.827343 16642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:41:33.832846 16642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:34979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=46181
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:33.833942 16642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:33.834934 16642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:33.844270 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.844283 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:33.844285 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:33.844373 16642 server_base.cc:1034] running on GCE node
I20260925 16:41:33.844921 16642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:33.845626 16642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:33.846804 16642 hybrid_clock.cc:648] HybridClock initialized: now 1790354493846771 us; error 43 us; skew 500 ppm
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:33.849532 16642 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:41:33.850351 16642 webserver.cc:460] Webserver started at http://127.0.97.195:46181/ using document root <none> and password file <none>
I20260925 16:41:33.850906 16642 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:33.851011 16642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:33.854821 16642 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 16:41:33.856783 16657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.858014 16642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:41:33.858152 16642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "f7af947022f04ea0acd871dba14be977"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "ab0b6a0d0abbf3058adaa23da2e3e5d4"
server_key_iv: "d4d85fc8ad7bc0a931d8fb7f38f2b67f"
server_key_version: "encryptionkey@0"
I20260925 16:41:33.858690 16642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:33.869539 16642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:33.872180 16642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:33.872381 16642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:33.872923 16642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:33.873747 16642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:33.873819 16642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.873885 16642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:33.873935 16642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:33.882707 16642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:34979
I20260925 16:41:33.882727 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:34979 every 8 connection(s)
I20260925 16:41:33.883643 16642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:41:33.889009 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16642
Sep 25 16:41:33 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:41:33.897527 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.889649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.97.254:34289)
0925 16:41:33.889894 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.889898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.890025 (+ 127us) server_negotiation.cc:407] Connection header received
0925 16:41:33.890103 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.890106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.890150 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.890305 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.890985 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.891512 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.892346 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.892569 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.892672 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.894182 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.894188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.894190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.894244 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.895861 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.896331 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.896334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.896335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.896406 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.896679 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.896682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.896683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.896860 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.897025 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.897190 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.897343 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260925 16:41:33.897748 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.885389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:53025 (local address 127.0.97.254:34289)
0925 16:41:33.885528 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.885533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.886352 (+ 819us) server_negotiation.cc:407] Connection header received
0925 16:41:33.887545 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.887548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.887594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.887669 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.888801 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.889477 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.890903 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.891153 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.891272 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.893769 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.893772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.893775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.893794 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.895592 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.896127 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.896130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.896131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.896206 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.896587 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.896589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.896590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.896806 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.896909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.897506 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.897615 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
I20260925 16:41:33.899008 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.885778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:53025)
0925 16:41:33.886188 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.886228 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.887355 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.887828 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.887834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.888169 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:33.888639 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.888650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.889609 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.889612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.890780 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.890789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.891306 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.891310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.891480 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.892064 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:33.892079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:33.893601 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:33.895702 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.895706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.895717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.896028 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.896312 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:33.896317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:33.896319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:33.896496 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:33.896927 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:33.896934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:33.897224 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.897467 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.898117 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260925 16:41:33.900168 16771 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34289
I20260925 16:41:33.900416 16771 heartbeater.cc:464] Registering TS with master...
I20260925 16:41:33.900875 16771 heartbeater.cc:511] Master 127.0.97.254:34289 requested a full tablet report, sending...
I20260925 16:41:33.901868 16313 ts_manager.cc:194] Registered new tserver with Master: f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979)
I20260925 16:41:33.902416 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:53025
I20260925 16:41:33.911075 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:41:33.914959 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.911737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.0.97.254:34289)
0925 16:41:33.911858 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.911862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.911999 (+ 137us) server_negotiation.cc:407] Connection header received
0925 16:41:33.912129 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.912131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.912225 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.912310 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:41:33.912519 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.913038 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.914006 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.914211 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.914291 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.914532 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:41:33.914660 (+ 128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 16:41:33.914861 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50004: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:41:33.915045 16504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50004: token expired
W20260925 16:41:33.915453 16238 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.254:34289)
I20260925 16:41:33.922683 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.915860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.254:34289)
0925 16:41:33.915986 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.915990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.916008 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:41:33.916097 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.916099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.916133 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.916198 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:33.916760 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.917225 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.918025 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.918229 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.918308 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.919647 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:33.919649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:33.919651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:33.919672 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:33.921130 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.921571 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.921574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.921575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.921671 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:33.921936 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:33.921939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:33.921940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:33.922104 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:33.922212 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.922392 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.922477 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:41:33.924978 16313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:41:33.928537 16313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:41:33.936662 16435 tablet_service.cc:1467] Processing CreateTablet for tablet 621d2e7770c04ab9abda890f52242f77 (DEFAULT_TABLE table=test-table [id=1ec8aa43d83f44b0a792a4b3664bbb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:33.937419 16435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621d2e7770c04ab9abda890f52242f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:33.940223 16393 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap starting.
I20260925 16:41:33.941092 16393 tablet_bootstrap.cc:654] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:33.941448 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.935342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.0.1:41886)
0925 16:41:33.935444 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.935461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.935543 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.936763 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.936766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.936784 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:33.936899 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.936907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.938517 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.938521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.939840 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.939851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.940870 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.940874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.940982 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.941050 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.941163 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.941255 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 16:41:33.941691 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.935680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.0.97.195:34979)
0925 16:41:33.936336 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.936345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.936367 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:41:33.936438 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.936442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.936616 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.936786 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:33.937116 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.938392 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.939955 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.940737 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.940857 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.940999 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.941210 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.941364 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":116,"threads_started":1}
I20260925 16:41:33.941896 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.935911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.0.97.194:39641)
0925 16:41:33.936419 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:33.936431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:33.936456 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:41:33.936531 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:33.936537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:33.936748 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:33.936911 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:33.937365 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.938457 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.939879 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.940735 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.940857 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.940999 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:33.941507 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 16:41:33.941746 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":119,"threads_started":1}
I20260925 16:41:33.942229 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:33.935662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.0.1:59650)
0925 16:41:33.935989 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:33.936005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:33.936097 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:33.937114 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:33.937117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:33.937140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:33.937224 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.937230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.938563 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.938566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.939748 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:33.939758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:33.941079 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:33.941083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:33.941205 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:33.941229 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:41:33.941369 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 16:41:33.941441 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":68,"thread_start_us":146,"threads_started":1}
I20260925 16:41:33.942991 16393 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: No bootstrap required, opened a new log
I20260925 16:41:33.943090 16393 ts_tablet_manager.cc:1397] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:41:33.943449 16393 raft_consensus.cc:348] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.943533 16393 raft_consensus.cc:374] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:33.943560 16393 raft_consensus.cc:729] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d4f49936d84059bf3e997b3a482fca, State: Initialized, Role: FOLLOWER
I20260925 16:41:33.943710 16393 consensus_queue.cc:260] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.943986 16393 ts_tablet_manager.cc:1428] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:41:33.944806 16705 tablet_service.cc:1467] Processing CreateTablet for tablet 621d2e7770c04ab9abda890f52242f77 (DEFAULT_TABLE table=test-table [id=1ec8aa43d83f44b0a792a4b3664bbb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:33.944806 16573 tablet_service.cc:1467] Processing CreateTablet for tablet 621d2e7770c04ab9abda890f52242f77 (DEFAULT_TABLE table=test-table [id=1ec8aa43d83f44b0a792a4b3664bbb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:41:33.945977 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621d2e7770c04ab9abda890f52242f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:33.945972 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621d2e7770c04ab9abda890f52242f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:41:33.954674 16784 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Bootstrap starting.
I20260925 16:41:33.955430 16785 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Bootstrap starting.
I20260925 16:41:33.957896 16784 tablet_bootstrap.cc:654] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:33.957897 16785 tablet_bootstrap.cc:654] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Neither blocks nor log segments found. Creating new log.
I20260925 16:41:33.958874 16785 log.cc:824] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:33.958875 16784 log.cc:824] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:33.960992 16784 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: No bootstrap required, opened a new log
I20260925 16:41:33.960997 16785 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: No bootstrap required, opened a new log
I20260925 16:41:33.961207 16784 ts_tablet_manager.cc:1397] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260925 16:41:33.961228 16785 ts_tablet_manager.cc:1397] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:41:33.964349 16785 raft_consensus.cc:348] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.964617 16785 raft_consensus.cc:374] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:33.964454 16784 raft_consensus.cc:348] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.964654 16785 raft_consensus.cc:729] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7af947022f04ea0acd871dba14be977, State: Initialized, Role: FOLLOWER
I20260925 16:41:33.964711 16784 raft_consensus.cc:374] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:41:33.964769 16784 raft_consensus.cc:729] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8bd5f05e76481681da6f16d6aab269, State: Initialized, Role: FOLLOWER
I20260925 16:41:33.965164 16784 consensus_queue.cc:260] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.965154 16785 consensus_queue.cc:260] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:33.965896 16771 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
I20260925 16:41:33.966060 16639 heartbeater.cc:503] Master 127.0.97.254:34289 was elected leader, sending a full tablet report...
I20260925 16:41:33.966194 16784 ts_tablet_manager.cc:1428] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260925 16:41:33.966701 16785 ts_tablet_manager.cc:1428] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
W20260925 16:41:33.978241 16502 tablet.cc:2369] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:41:34.134554 16772 tablet.cc:2369] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:41:34.199091 16640 tablet.cc:2369] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:41:34.307335 16788 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:34.307603 16788 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:34.308923 16788 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329)
I20260925 16:41:34.310921 16507 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:34.311136 16507 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:34.312212 16507 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
I20260925 16:41:34.313642 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.309168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40329 (local address 127.0.97.194:56343)
0925 16:41:34.309569 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.309586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:34.309704 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:34.309986 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:34.309989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:34.310004 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:34.310146 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.310152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.310959 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.310963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.312084 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.312094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.313102 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.313106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.313289 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.313312 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:41:34.313388 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:41:34.313461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":42,"thread_start_us":93,"threads_started":1}
I20260925 16:41:34.313746 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.309251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:58725 (local address 127.0.97.195:34979)
0925 16:41:34.309420 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:34.309424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:34.309612 (+ 188us) server_negotiation.cc:407] Connection header received
0925 16:41:34.309738 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:34.309743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:34.309802 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:34.309900 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:34.310275 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.310767 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.312012 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.312597 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.312714 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.312790 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:34.313525 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 16:41:34.313603 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:41:34.314061 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.309331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:56343 (local address 127.0.97.193:40329)
0925 16:41:34.309593 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:34.309597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:34.309750 (+ 153us) server_negotiation.cc:407] Connection header received
0925 16:41:34.309806 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:34.309808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:34.309861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:34.309977 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:34.310265 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.310824 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.312225 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.312859 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.312960 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.313109 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:34.313862 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 16:41:34.313942 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260925 16:41:34.314112 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.309132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.97.194:58725)
0925 16:41:34.309488 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.309503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:34.309629 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:34.309922 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:34.309925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:34.309940 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:34.310108 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.310114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.310879 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.310885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.311887 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.311895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.312983 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.312986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.313105 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.313408 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 16:41:34.313909 (+ 501us) client_negotiation.cc:240] Negotiation successful
0925 16:41:34.313978 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":317,"thread_start_us":127,"threads_started":1}
I20260925 16:41:34.314476 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "db8bd5f05e76481681da6f16d6aab269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d4f49936d84059bf3e997b3a482fca" is_pre_election: true
I20260925 16:41:34.314713 16455 raft_consensus.cc:2454] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8bd5f05e76481681da6f16d6aab269 in term 0.
I20260925 16:41:34.314874 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "db8bd5f05e76481681da6f16d6aab269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7af947022f04ea0acd871dba14be977" is_pre_election: true
I20260925 16:41:34.315125 16725 raft_consensus.cc:2454] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8bd5f05e76481681da6f16d6aab269 in term 0.
I20260925 16:41:34.315147 16526 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca, db8bd5f05e76481681da6f16d6aab269; no voters:
I20260925 16:41:34.315411 16788 raft_consensus.cc:2781] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:41:34.315492 16788 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:41:34.315543 16788 raft_consensus.cc:3037] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:34.316882 16788 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:34.317188 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.312896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:33727 (local address 127.0.97.194:39641)
0925 16:41:34.312994 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:34.312998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:34.313069 (+ 71us) server_negotiation.cc:407] Connection header received
0925 16:41:34.313211 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:34.313215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:34.313255 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:34.313340 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:34.313581 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.314265 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.315938 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.316487 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.316597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.316637 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:34.317013 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 16:41:34.317080 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 16:41:34.317235 16788 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [CANDIDATE]: Term 1 election: Requested vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329)
I20260925 16:41:34.317438 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.312487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.97.193:33727)
0925 16:41:34.312996 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.313010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:34.313131 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:34.313355 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:34.313358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:34.313373 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:34.313467 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.313475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.314611 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.314614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.315808 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.315818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.316699 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.316705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.316819 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.316850 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:41:34.316967 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 16:41:34.317101 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":22,"thread_start_us":49,"threads_started":1}
I20260925 16:41:34.317656 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "db8bd5f05e76481681da6f16d6aab269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7af947022f04ea0acd871dba14be977"
I20260925 16:41:34.317791 16725 raft_consensus.cc:3037] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:34.317718 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.312339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.97.193:34767)
0925 16:41:34.312788 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.312801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:34.312884 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:34.313546 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:34.313549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:34.313566 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:41:34.313644 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.313652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.315537 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.315541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.316426 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:34.316434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.317314 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.317318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:34.317411 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.317433 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:41:34.317488 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 16:41:34.317553 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":49,"thread_start_us":54,"threads_started":1}
I20260925 16:41:34.317763 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "db8bd5f05e76481681da6f16d6aab269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d4f49936d84059bf3e997b3a482fca"
I20260925 16:41:34.318140 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.312601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34767 (local address 127.0.97.195:34979)
0925 16:41:34.313319 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:34.313328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:34.313342 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:41:34.313386 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:34.313388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:34.313431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:34.313755 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:41:34.313858 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.315403 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.316548 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.317127 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.317845 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.317896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:34.317931 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 16:41:34.317998 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":143,"threads_started":1}
I20260925 16:41:34.318167 16455 raft_consensus.cc:3037] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:41:34.318287 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8bd5f05e76481681da6f16d6aab269" is_pre_election: true
I20260925 16:41:34.318490 16593 raft_consensus.cc:2379] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10d4f49936d84059bf3e997b3a482fca in current term 1: Already voted for candidate db8bd5f05e76481681da6f16d6aab269 in this term.
I20260925 16:41:34.318921 16725 raft_consensus.cc:2454] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8bd5f05e76481681da6f16d6aab269 in term 1.
I20260925 16:41:34.319248 16529 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977; no voters:
I20260925 16:41:34.319413 16788 raft_consensus.cc:2781] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:41:34.319401 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621d2e7770c04ab9abda890f52242f77" candidate_uuid: "10d4f49936d84059bf3e997b3a482fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7af947022f04ea0acd871dba14be977" is_pre_election: true
I20260925 16:41:34.319463 16455 raft_consensus.cc:2454] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8bd5f05e76481681da6f16d6aab269 in term 1.
I20260925 16:41:34.319561 16725 raft_consensus.cc:2379] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10d4f49936d84059bf3e997b3a482fca in current term 1: Already voted for candidate db8bd5f05e76481681da6f16d6aab269 in this term.
I20260925 16:41:34.319625 16788 raft_consensus.cc:686] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 LEADER]: Becoming Leader. State: Replica: db8bd5f05e76481681da6f16d6aab269, State: Running, Role: LEADER
I20260925 16:41:34.319977 16788 consensus_queue.cc:237] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:34.320046 16390 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca; no voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977
I20260925 16:41:34.320313 16507 raft_consensus.cc:2726] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 16:41:34.323016 16313 catalog_manager.cc:5495] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 reported cstate change: term changed from 0 to 1, leader changed from <none> to db8bd5f05e76481681da6f16d6aab269 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "db8bd5f05e76481681da6f16d6aab269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } health_report { overall_health: UNKNOWN } } }
I20260925 16:41:34.395449 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.391781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.0.97.194:39641)
0925 16:41:34.391937 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:34.391943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:34.391959 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:41:34.392045 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:34.392047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:34.392098 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:34.392214 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:41:34.392458 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.392957 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.393994 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:34.394208 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:34.394327 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:34.394649 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:41:34.395061 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:41:34.395198 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:34.395239 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 16:41:34.395298 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:41:34.401495 16725 raft_consensus.cc:1261] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Refusing update from remote peer db8bd5f05e76481681da6f16d6aab269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:41:34.401552 16455 raft_consensus.cc:1261] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Refusing update from remote peer db8bd5f05e76481681da6f16d6aab269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:41:34.402138 16796 consensus_queue.cc:1035] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:41:34.402380 16788 consensus_queue.cc:1035] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:41:34.407281 16799 mvcc.cc:205] Tried to move back new op lower bound from 7333292009060675584 to 7333292008737394688. Current Snapshot: MvccSnapshot[applied={T|T < 7333292009060675584}]
I20260925 16:41:34.411089 16801 mvcc.cc:205] Tried to move back new op lower bound from 7333292009060675584 to 7333292008737394688. Current Snapshot: MvccSnapshot[applied={T|T < 7333292009060675584}]
I20260925 16:41:34.417104 16313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:41:34.417264 16313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:41:34.419606 16313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Sending DeleteTablet for 3 replicas of tablet 621d2e7770c04ab9abda890f52242f77
I20260925 16:41:34.420357 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet 621d2e7770c04ab9abda890f52242f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:34 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:41886
I20260925 16:41:34.420357 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet 621d2e7770c04ab9abda890f52242f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:41:34 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:59650
I20260925 16:41:34.420459 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16351
I20260925 16:41:34.421512 16809 tablet_replica.cc:330] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: stopping tablet replica
I20260925 16:41:34.421861 16809 raft_consensus.cc:2229] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:41:34.422165 16809 raft_consensus.cc:2258] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:41:34.422506 16808 tablet_replica.cc:330] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: stopping tablet replica
I20260925 16:41:34.422809 16808 raft_consensus.cc:2229] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:41:34.423192 16808 raft_consensus.cc:2258] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:41:34.423650 16809 ts_tablet_manager.cc:1905] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:41:34.424526 16808 ts_tablet_manager.cc:1905] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:41:34.426294 16809 ts_tablet_manager.cc:1918] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:41:34.426403 16809 log.cc:1196] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/621d2e7770c04ab9abda890f52242f77
I20260925 16:41:34.426738 16809 ts_tablet_manager.cc:1939] T 621d2e7770c04ab9abda890f52242f77 P f7af947022f04ea0acd871dba14be977: Deleting consensus metadata
I20260925 16:41:34.428042 16808 ts_tablet_manager.cc:1918] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:41:34.428141 16808 log.cc:1196] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/621d2e7770c04ab9abda890f52242f77
I20260925 16:41:34.428099 16300 catalog_manager.cc:4841] TS f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): tablet 621d2e7770c04ab9abda890f52242f77 (table test-table [id=1ec8aa43d83f44b0a792a4b3664bbb85]) successfully deleted
I20260925 16:41:34.428522 16808 ts_tablet_manager.cc:1939] T 621d2e7770c04ab9abda890f52242f77 P db8bd5f05e76481681da6f16d6aab269: Deleting consensus metadata
W20260925 16:41:34.428517 16297 connection.cc:535] client connection to 127.0.97.193:40329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 16:41:34.428678 16297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:41:34.429035 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16505
I20260925 16:41:34.434058 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.433804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:40329 (local address 127.0.0.1:60674)
0925 16:41:34.433930 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.433988 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 16:41:34.434221 16297 catalog_manager.cc:4584] TS 10d4f49936d84059bf3e997b3a482fca (127.0.97.193:40329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 621d2e7770c04ab9abda890f52242f77: Network error: Client connection negotiation failed: client connection to 127.0.97.193:40329: connect: Connection refused (error 111)
I20260925 16:41:34.435010 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16642
I20260925 16:41:34.436893 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:34.435195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.0.1:59666)
0925 16:41:34.436734 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:34.436800 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:39641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1442}
I20260925 16:41:34.440511 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16280
I20260925 16:41:54.446977 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--webserver_interface=127.0.97.254
--webserver_port=33909
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:54.563297 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:54.563584 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:54.563669 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:54.567728 16811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:41:54.567785 16811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:41:54.567799 16811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:41:54.567813 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:41:54.567832 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:54.567899 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:54.567929 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:41:54.567957 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:41:54.573444 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:54.574519 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:54.575603 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:54.584259 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:54.584302 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:54.584261 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:54.584625 16811 server_base.cc:1034] running on GCE node
I20260925 16:41:54.585194 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:54.586293 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:54.587491 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790354514587463 us; error 30 us; skew 500 ppm
Sep 25 16:41:54 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:54.590374 16811 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:41:54.591277 16811 webserver.cc:460] Webserver started at http://127.0.97.254:33909/ using document root <none> and password file <none>
I20260925 16:41:54.591790 16811 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:54.591861 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:54.595726 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:41:54.597855 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:54.599035 16811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:41:54.599133 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "debca3c9b52e47438b44ef04cd1eb2b5"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "b0735fa6dab5eab32b958f7c74703a69"
server_key_iv: "04214a71bf3eae1365e197abe9712577"
server_key_version: "encryptionkey@0"
I20260925 16:41:54.599696 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:54.615312 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:54.618250 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:54.618402 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:54.625465 16811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:34289
I20260925 16:41:54.625762 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:34289 every 8 connection(s)
I20260925 16:41:54.626550 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:41:54.630692 16879 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:41:54.632246 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap starting.
I20260925 16:41:54.633575 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16811
I20260925 16:41:54.634035 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:40329
--local_ip_for_outbound_sockets=127.0.97.193
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=43981
--webserver_interface=127.0.97.193
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:41:54.650030 16879 log.cc:824] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:54.662190 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:41:54.662611 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Bootstrap complete.
I20260925 16:41:54.666656 16879 raft_consensus.cc:348] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:54.667032 16879 raft_consensus.cc:729] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Initialized, Role: FOLLOWER
I20260925 16:41:54.667522 16879 consensus_queue.cc:260] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:54.667634 16879 raft_consensus.cc:388] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:41:54.667718 16879 raft_consensus.cc:482] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:41:54.667817 16879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:41:54.669734 16879 raft_consensus.cc:504] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:54.670058 16879 leader_election.cc:302] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debca3c9b52e47438b44ef04cd1eb2b5; no voters:
I20260925 16:41:54.670423 16879 leader_election.cc:288] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:41:54.670504 16883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:41:54.670686 16883 raft_consensus.cc:686] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [term 3 LEADER]: Becoming Leader. State: Replica: debca3c9b52e47438b44ef04cd1eb2b5, State: Running, Role: LEADER
I20260925 16:41:54.671049 16883 consensus_queue.cc:237] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } }
I20260925 16:41:54.671681 16879 sys_catalog.cc:564] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:41:54.672037 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debca3c9b52e47438b44ef04cd1eb2b5. Latest consensus state: current_term: 3 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:54.672147 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:54.672505 16887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:41:54.672809 16884 sys_catalog.cc:455] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debca3c9b52e47438b44ef04cd1eb2b5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34289 } } }
I20260925 16:41:54.673108 16884 sys_catalog.cc:458] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:41:54.677737 16887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:41:54.678388 16887 catalog_manager.cc:1259] Loaded cluster ID: 3fa5f15e1840446c83e38e8586af5925
I20260925 16:41:54.678448 16887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:41:54.681012 16887 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:41:54.681839 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Loaded TSK: 0
I20260925 16:41:54.682094 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P debca3c9b52e47438b44ef04cd1eb2b5: Loaded TSK: 1
F20260925 16:41:54.682914 16887 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f069c1e29ad google::LogMessage::Fail() at ??:0
@ 0x7f069c1e588a google::LogMessage::SendToLog() at ??:0
@ 0x7f069c1e24de google::LogMessage::Flush() at ??:0
@ 0x7f069c1e3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f069eea618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f069ee9ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f06a6b477bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f06a6b2199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f06a6b22178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f06a6b63df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f06a6b8282b std::function<>::operator()() at ??:0
@ 0x7f06a6b21b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f06a6b22f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f06a6b1f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f06a6b63640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f069e4343d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f069e434c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f069e4363f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f069e425512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f069fa696db start_thread at ??:0
@ 0x7f069ace061f clone at ??:0
W20260925 16:41:54.762360 16881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:54.762639 16881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:54.762686 16881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:54.766949 16881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:54.766992 16881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:54.767050 16881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:41:54.773180 16881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=43981
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:54.774443 16881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:54.775503 16881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:54.785560 16926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:54.785575 16928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:54.785856 16881 server_base.cc:1034] running on GCE node
W20260925 16:41:54.786083 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:54.786461 16881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:54.787226 16881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:54.788407 16881 hybrid_clock.cc:648] HybridClock initialized: now 1790354514788374 us; error 55 us; skew 500 ppm
Sep 25 16:41:54 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:54.791283 16881 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:41:54.792147 16881 webserver.cc:460] Webserver started at http://127.0.97.193:43981/ using document root <none> and password file <none>
I20260925 16:41:54.792730 16881 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:54.792820 16881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:54.797086 16881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 16:41:54.799377 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:54.800709 16881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:54.800835 16881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "10d4f49936d84059bf3e997b3a482fca"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "5e76d5b30e2c5da1f9bf75038965dd23"
server_key_iv: "6f9b9a956797aa0981d1927176cbb7cf"
server_key_version: "encryptionkey@0"
I20260925 16:41:54.801412 16881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:54.813258 16881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:54.816355 16881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:54.816550 16881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:54.817104 16881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:54.818220 16943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:41:54.823522 16881 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:41:54.823611 16881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 16:41:54.823693 16881 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:41:54.825265 16881 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:41:54.825390 16881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:54.825704 16943 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap starting.
I20260925 16:41:54.835923 16881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:40329
I20260925 16:41:54.836118 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:40329 every 8 connection(s)
I20260925 16:41:54.836862 16881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:41:54.840806 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 16881
I20260925 16:41:54.841204 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:39641
--local_ip_for_outbound_sockets=127.0.97.194
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=34385
--webserver_interface=127.0.97.194
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:41:54.847852 16943 log.cc:824] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Log is configured to *not* fsync() on all Append() calls
I20260925 16:41:54.856665 16943 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:41:54.857075 16943 tablet_bootstrap.cc:492] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Bootstrap complete.
I20260925 16:41:54.858536 16943 ts_tablet_manager.cc:1397] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.014s
I20260925 16:41:54.861706 16943 raft_consensus.cc:348] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:54.862259 16943 raft_consensus.cc:729] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d4f49936d84059bf3e997b3a482fca, State: Initialized, Role: FOLLOWER
I20260925 16:41:54.862844 16943 consensus_queue.cc:260] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:54.863915 16943 ts_tablet_manager.cc:1428] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 16:41:54.928025 17053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:41:54.838807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:49305)
0925 16:41:54.839282 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:54.839335 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:54.840134 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:54.926959 (+ 86825us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
W20260925 16:41:54.928829 17051 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:34289 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:34289: Client connection negotiation failed: client connection to 127.0.97.254:34289: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:41:54.929862 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:54.929452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:58143)
0925 16:41:54.929596 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:54.929673 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 16:41:54.930619 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:54.930391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:48095)
0925 16:41:54.930508 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:54.930561 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:41:54.930821 17051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:41:54.959503 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:54.959750 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:54.959838 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:54.963964 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:54.964025 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:54.964136 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:41:54.969411 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:39641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=34385
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:54.970458 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:54.971504 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:54.981534 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:54.981513 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:54.981513 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:54.981807 17054 server_base.cc:1034] running on GCE node
I20260925 16:41:54.982347 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:54.983035 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:54.984222 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790354514984199 us; error 36 us; skew 500 ppm
Sep 25 16:41:54 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:54.987080 17054 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:41:54.987871 17054 webserver.cc:460] Webserver started at http://127.0.97.194:34385/ using document root <none> and password file <none>
I20260925 16:41:54.988385 17054 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:54.988442 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:54.992130 17054 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:54.994195 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:54.995481 17054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:54.995596 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "db8bd5f05e76481681da6f16d6aab269"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "d1dc35243504434601812e070d66a1db"
server_key_iv: "6db0edc55c79a014017d7abcaa4a1396"
server_key_version: "encryptionkey@0"
I20260925 16:41:54.996106 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:55.042027 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:55.045468 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:55.045619 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:55.046260 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:55.047082 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:55.047132 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:55.047197 17054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:55.047256 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:55.055842 17054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:39641
I20260925 16:41:55.055873 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:39641 every 8 connection(s)
I20260925 16:41:55.056802 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:41:55.060510 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.058826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:40505)
0925 16:41:55.059260 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.059366 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
W20260925 16:41:55.060956 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:34289 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:34289: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
I20260925 16:41:55.061586 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.061375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:58107)
0925 16:41:55.061486 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.061537 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 16:41:55.062139 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.061947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:54175)
0925 16:41:55.062044 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.062086 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 16:41:55.062297 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:41:55.062964 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17054
I20260925 16:41:55.063360 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:34979
--local_ip_for_outbound_sockets=127.0.97.195
--tserver_master_addrs=127.0.97.254:34289
--webserver_port=46181
--webserver_interface=127.0.97.195
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:41:55.171350 17188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:41:55.171651 17188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:41:55.171733 17188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:41:55.175870 17188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:41:55.175963 17188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:41:55.176056 17188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:41:55.181197 17188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:34979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=46181
--tserver_master_addrs=127.0.97.254:34289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:41:55.182363 17188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:41:55.183415 17188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:41:55.192839 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:55.192838 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:41:55.192925 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:41:55.193054 17188 server_base.cc:1034] running on GCE node
I20260925 16:41:55.193467 17188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:41:55.194108 17188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:41:55.195284 17188 hybrid_clock.cc:648] HybridClock initialized: now 1790354515195271 us; error 26 us; skew 500 ppm
Sep 25 16:41:55 dist-test-slave-kxl8 krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:41:55.197991 17188 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:41:55.198979 17188 webserver.cc:460] Webserver started at http://127.0.97.195:46181/ using document root <none> and password file <none>
I20260925 16:41:55.199535 17188 fs_manager.cc:359] Metadata directory not provided
I20260925 16:41:55.199620 17188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:41:55.203241 17188 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:41:55.205163 17203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:55.206439 17188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:41:55.206561 17188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "f7af947022f04ea0acd871dba14be977"
format_stamp: "Formatted at 2026-09-25 16:41:12 on dist-test-slave-kxl8"
server_key: "ab0b6a0d0abbf3058adaa23da2e3e5d4"
server_key_iv: "d4d85fc8ad7bc0a931d8fb7f38f2b67f"
server_key_version: "encryptionkey@0"
I20260925 16:41:55.207084 17188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:41:55.225247 17188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:41:55.228041 17188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:41:55.228242 17188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:41:55.228818 17188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:41:55.229661 17188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:41:55.229730 17188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:55.229801 17188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:41:55.229851 17188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:41:55.238786 17188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:34979
I20260925 16:41:55.238821 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:34979 every 8 connection(s)
I20260925 16:41:55.239786 17188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:41:55.243624 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.241941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:44339)
0925 16:41:55.242382 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.242489 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
W20260925 16:41:55.244061 17317 heartbeater.cc:644] Failed to heartbeat to 127.0.97.254:34289 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.254:34289: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
I20260925 16:41:55.244674 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.244453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:50519)
0925 16:41:55.244567 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.244617 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 16:41:55.245225 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.245028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:37531)
0925 16:41:55.245130 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.245173 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:41:55.245407 17317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:41:55.249229 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17188
I20260925 16:41:55.931860 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:55.931443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:34275)
0925 16:41:55.931696 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:55.931752 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260925 16:41:56.063244 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.062842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:43555)
0925 16:41:56.063089 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:56.063145 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260925 16:41:56.219753 17323 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:56.219957 17323 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:56.220957 17323 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
Sep 25 16:41:56 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.195@KRBTEST.COM
Sep 25 16:41:56 dist-test-slave-kxl8 krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.194@KRBTEST.COM
I20260925 16:41:56.234725 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.221194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:34979 (local address 127.0.97.193:55029)
0925 16:41:56.221326 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:56.221342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:56.221464 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:56.223155 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:56.223161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:56.223680 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:56.224156 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:56.224169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.225496 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.225499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:56.226346 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:56.226354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.227103 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.227108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:56.227217 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:56.227992 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:56.228007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:56.229601 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:56.231997 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:56.232012 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:56.232313 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:56.232687 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:56.232691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:56.232861 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:56.233613 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:56.233619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:56.233954 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 16:41:56.234168 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 16:41:56.234518 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":208,"spinlock_wait_cycles":7680}
I20260925 16:41:56.234772 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.221378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:55029 (local address 127.0.97.195:34979)
0925 16:41:56.221732 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:56.221749 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:56.221797 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:41:56.222468 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:56.222474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:56.222807 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:56.223096 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:56.224329 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.225335 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.226471 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.226961 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.227593 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:56.229823 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:56.229828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:56.229839 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:56.229867 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:56.231873 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232429 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:56.232433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:56.232437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:56.232592 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232971 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:56.232974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:56.232976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:56.233492 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:56.233627 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:56.234303 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 16:41:56.234585 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":110,"threads_started":1}
I20260925 16:41:56.234726 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.221194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:39641 (local address 127.0.97.193:50947)
0925 16:41:56.221661 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:56.221675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:41:56.221758 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:41:56.223157 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:41:56.223162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:41:56.223679 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:41:56.224156 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:56.224170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.225437 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.225444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:56.226344 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:41:56.226353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.227059 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.227063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:41:56.227230 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:56.228039 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:41:56.228050 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:41:56.229950 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:41:56.232234 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:56.232239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:56.232471 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:56.232802 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:41:56.232806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:41:56.232877 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:41:56.233613 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:41:56.233619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:41:56.233954 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 16:41:56.234168 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 16:41:56.234570 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":355,"spinlock_wait_cycles":62208,"thread_start_us":72,"threads_started":1}
I20260925 16:41:56.234772 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.221378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:50947 (local address 127.0.97.194:39641)
0925 16:41:56.221788 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 16:41:56.221806 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:41:56.221839 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:41:56.222470 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:41:56.222476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:41:56.222806 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:41:56.223109 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:41:56.224333 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.225256 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.226474 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:41:56.226933 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:41:56.227093 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:41:56.230133 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:41:56.230142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:41:56.230158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:41:56.230194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:41:56.232132 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232568 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:56.232573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:56.232577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:56.232710 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:41:56.232971 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:41:56.232974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:41:56.232976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:41:56.233492 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:41:56.233627 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:41:56.234303 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 16:41:56.234585 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":103,"threads_started":1}
W20260925 16:41:56.237486 16937 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
W20260925 16:41:56.237663 16940 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
I20260925 16:41:56.237746 16940 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca; no voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977
I20260925 16:41:56.238024 17323 raft_consensus.cc:2726] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:41:56.246356 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.245947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:54365)
0925 16:41:56.246214 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:56.246266 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":111,"threads_started":1}
I20260925 16:41:56.933089 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:56.932698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:34387)
0925 16:41:56.932940 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:56.932996 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260925 16:41:57.064280 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:57.063921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:35733)
0925 16:41:57.064147 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:57.064198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260925 16:41:57.247617 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:57.247192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:35961)
0925 16:41:57.247448 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:57.247507 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260925 16:41:57.934322 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:57.933871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:60881)
0925 16:41:57.934162 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:57.934228 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260925 16:41:58.065478 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:58.065018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:54985)
0925 16:41:58.065263 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:58.065372 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260925 16:41:58.204521 17333 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:41:58.204645 17333 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:41:58.205135 17333 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
W20260925 16:41:58.205974 16937 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
W20260925 16:41:58.206063 16940 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
I20260925 16:41:58.206178 16940 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca; no voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977
I20260925 16:41:58.206360 17333 raft_consensus.cc:2726] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:41:58.248790 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:58.248369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:43287)
0925 16:41:58.248635 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:58.248697 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260925 16:41:58.935590 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:58.935144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:49561)
0925 16:41:58.935431 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:58.935494 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260925 16:41:59.066748 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:59.066228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:45883)
0925 16:41:59.066589 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:59.066653 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":88,"threads_started":1}
I20260925 16:41:59.250005 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:59.249581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:35859)
0925 16:41:59.249848 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:59.249908 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260925 16:41:59.936892 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:41:59.936465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:43741)
0925 16:41:59.936736 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 16:41:59.936788 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260925 16:42:00.067934 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:00.067542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:42753)
0925 16:42:00.067787 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:00.067845 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260925 16:42:00.251312 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:00.250875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:38553)
0925 16:42:00.251153 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:00.251218 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260925 16:42:00.938262 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:00.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:37803)
0925 16:42:00.938062 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:00.938120 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
I20260925 16:42:01.068960 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:01.068582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:39993)
0925 16:42:01.068818 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:01.068873 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260925 16:42:01.096841 17343 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:01.097002 17343 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:42:01.097486 17343 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
W20260925 16:42:01.098276 16937 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
W20260925 16:42:01.098372 16940 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
I20260925 16:42:01.098476 16940 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca; no voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977
I20260925 16:42:01.098639 17343 raft_consensus.cc:2726] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:42:01.252406 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:01.252021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:59055)
0925 16:42:01.252253 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:01.252306 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260925 16:42:01.939473 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:01.939077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:58883)
0925 16:42:01.939320 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:01.939379 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260925 16:42:02.070215 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:02.069787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:48289)
0925 16:42:02.070070 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:02.070125 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260925 16:42:02.253679 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:02.253158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:51167)
0925 16:42:02.253523 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:02.253581 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260925 16:42:02.940619 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:02.940226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:46411)
0925 16:42:02.940470 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:02.940526 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260925 16:42:03.071447 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:03.071039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:52565)
0925 16:42:03.071298 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:03.071356 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
I20260925 16:42:03.254958 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:03.254536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:60953)
0925 16:42:03.254793 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:03.254856 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260925 16:42:03.941962 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:03.941549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:40405)
0925 16:42:03.941804 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:03.941864 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260925 16:42:04.072654 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:04.072224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:57385)
0925 16:42:04.072492 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:04.072554 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260925 16:42:04.256265 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:04.255761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:44811)
0925 16:42:04.256105 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:04.256170 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260925 16:42:04.943228 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:04.942784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:36947)
0925 16:42:04.943071 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:04.943130 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260925 16:42:05.073830 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:05.073415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:55849)
0925 16:42:05.073680 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:05.073738 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260925 16:42:05.257608 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:05.257135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:51801)
0925 16:42:05.257454 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:05.257512 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":87,"threads_started":1}
I20260925 16:42:05.392128 17357 raft_consensus.cc:482] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:05.392280 17357 raft_consensus.cc:504] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7af947022f04ea0acd871dba14be977" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 34979 } } peers { permanent_uuid: "db8bd5f05e76481681da6f16d6aab269" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 39641 } } peers { permanent_uuid: "10d4f49936d84059bf3e997b3a482fca" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 40329 } }
I20260925 16:42:05.392761 17357 leader_election.cc:288] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979), db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641)
W20260925 16:42:05.393502 16940 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7af947022f04ea0acd871dba14be977 (127.0.97.195:34979): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
W20260925 16:42:05.393599 16937 leader_election.cc:341] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db8bd5f05e76481681da6f16d6aab269 (127.0.97.194:39641): Not found: Tablet not found: 621d2e7770c04ab9abda890f52242f77
I20260925 16:42:05.393781 16937 leader_election.cc:302] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10d4f49936d84059bf3e997b3a482fca; no voters: db8bd5f05e76481681da6f16d6aab269, f7af947022f04ea0acd871dba14be977
I20260925 16:42:05.393923 17357 raft_consensus.cc:2726] T 621d2e7770c04ab9abda890f52242f77 P 10d4f49936d84059bf3e997b3a482fca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:42:05.944416 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:05.944013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:51105)
0925 16:42:05.944250 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:05.944303 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260925 16:42:06.075002 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:06.074599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:41839)
0925 16:42:06.074849 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:06.074908 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260925 16:42:06.258955 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:06.258494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:40191)
0925 16:42:06.258776 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:06.258839 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260925 16:42:06.945765 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:06.945257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:47873)
0925 16:42:06.945609 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:06.945668 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":99,"threads_started":1}
I20260925 16:42:07.076140 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:07.075716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:39633)
0925 16:42:07.075988 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:07.076046 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260925 16:42:07.260231 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:07.259829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:34909)
0925 16:42:07.260079 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:07.260136 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260925 16:42:07.947021 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:07.946575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:53559)
0925 16:42:07.946834 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:07.946896 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260925 16:42:08.077548 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:08.076983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:41993)
0925 16:42:08.077403 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:08.077458 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":102,"threads_started":1}
I20260925 16:42:08.261585 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:08.261136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:33551)
0925 16:42:08.261437 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:08.261490 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260925 16:42:08.948163 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:08.947770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:42201)
0925 16:42:08.948019 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:08.948071 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260925 16:42:09.078660 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:09.078247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:39149)
0925 16:42:09.078507 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:09.078562 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260925 16:42:09.262903 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:09.262447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:48035)
0925 16:42:09.262744 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:09.262805 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
I20260925 16:42:09.949633 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:09.949146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.193:55323)
0925 16:42:09.949475 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:09.949538 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":86,"threads_started":1}
I20260925 16:42:10.080077 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:10.079592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.194:36919)
0925 16:42:10.079916 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:10.079978 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
I20260925 16:42:10.255563 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 16881
I20260925 16:42:10.261473 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17054
I20260925 16:42:10.264379 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:10.263916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34289 (local address 127.0.97.195:41919)
0925 16:42:10.264198 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:10.264256 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.254:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260925 16:42:10.267217 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17188
2026-09-25T16:42:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59451 ms)
[----------] 30 tests from SecurityITest (241291 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:10 dist-test-slave-kxl8 krb5kdc[17375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:10 dist-test-slave-kxl8 krb5kdc[17375](info): set up 2 sockets
Sep 25 16:42:10 dist-test-slave-kxl8 krb5kdc[17375](info): commencing operation
krb5kdc: starting...
W20260925 16:42:11.313079 391 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:11 dist-test-slave-kxl8 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:11.436496 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44523
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:11.545606 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:11.545881 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:11.545964 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:11.550057 17388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:11.550120 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:11.550139 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:11.550194 17388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:11.550221 17388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:11.555660 17388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:11.556748 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:11.557837 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:11.566658 17388 server_base.cc:1034] running on GCE node
W20260925 16:42:11.566603 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:11.566581 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:11.566594 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:11.567387 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:11.568502 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:11.569658 17388 hybrid_clock.cc:648] HybridClock initialized: now 1790354531569641 us; error 30 us; skew 500 ppm
Sep 25 16:42:11 dist-test-slave-kxl8 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:11.572367 17388 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:11.573201 17388 webserver.cc:460] Webserver started at http://127.0.97.254:40505/ using document root <none> and password file <none>
I20260925 16:42:11.573774 17388 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:11.573868 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:11.574095 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:11.576048 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "6b7417b7309149ee87fe263920216b6f"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "97b0ee8e217515694392ebd19dffd996"
server_key_iv: "d584895a668c66d09b066789aa6f3eb3"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.576606 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "6b7417b7309149ee87fe263920216b6f"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "97b0ee8e217515694392ebd19dffd996"
server_key_iv: "d584895a668c66d09b066789aa6f3eb3"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.580144 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:42:11.582473 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:11.583636 17388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:11.583781 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "6b7417b7309149ee87fe263920216b6f"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "97b0ee8e217515694392ebd19dffd996"
server_key_iv: "d584895a668c66d09b066789aa6f3eb3"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.583892 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:11.642418 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:11.646001 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:11.646221 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:11.653018 17388 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44523
I20260925 16:42:11.653074 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44523 every 8 connection(s)
I20260925 16:42:11.654024 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:11.656795 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:11.662171 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: Bootstrap starting.
I20260925 16:42:11.662883 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17388
I20260925 16:42:11.663012 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:11.663298 391 external_mini_cluster.cc:1428] Setting key bd9ac4a40b5f3f4369b8c1fbb7d5f3bc
I20260925 16:42:11.664983 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:11.665884 17456 log.cc:824] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:11.668044 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: No bootstrap required, opened a new log
I20260925 16:42:11.672039 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } }
I20260925 16:42:11.672364 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:11.672451 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7417b7309149ee87fe263920216b6f, State: Initialized, Role: FOLLOWER
Sep 25 16:42:11 dist-test-slave-kxl8 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:11.672943 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } }
I20260925 16:42:11.673117 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:11.673187 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:11.673319 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:11.674199 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } }
I20260925 16:42:11.674535 17456 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7417b7309149ee87fe263920216b6f; no voters:
I20260925 16:42:11.674860 17456 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:11.674963 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:11.675139 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [term 1 LEADER]: Becoming Leader. State: Replica: 6b7417b7309149ee87fe263920216b6f, State: Running, Role: LEADER
I20260925 16:42:11.675475 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } }
I20260925 16:42:11.676139 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:11.676620 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7417b7309149ee87fe263920216b6f. Latest consensus state: current_term: 1 leader_uuid: "6b7417b7309149ee87fe263920216b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } } }
I20260925 16:42:11.676808 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:11.677263 17467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:11.677681 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7417b7309149ee87fe263920216b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7417b7309149ee87fe263920216b6f" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44523 } } }
I20260925 16:42:11.677861 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:11.679770 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:11.664580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.0.97.254:44523)
0925 16:42:11.665038 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:11.665064 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:11.665109 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:42:11.665996 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:11.666019 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:11.666462 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:11.666822 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:11.667438 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.668901 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.669772 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.670470 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.670657 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:11.673426 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:11.673435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:11.673452 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:11.673484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:11.675517 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:11.675977 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:11.675982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:11.675986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:11.676087 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:11.676443 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:11.676447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:11.676448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:11.676963 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:11.677147 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:11.678546 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0925 16:42:11.678853 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":104,"threads_started":1}
I20260925 16:42:11.680570 17467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:11.687080 17467 catalog_manager.cc:1347] Generated new cluster ID: e0869cde7bda4601a39d71de3406bc98
I20260925 16:42:11.687191 17467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:11.698603 17467 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:11.699527 17467 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:11.703197 17467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: Generated new TSK 0
I20260925 16:42:11.703855 17467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:42:11.735891 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:11.862280 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:11.862555 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:11.862630 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:11.866601 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:11.866686 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:11.866776 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:11.871989 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:11.873003 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:11.874028 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:11.883560 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:11.883579 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:11.883579 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:11.883615 17482 server_base.cc:1034] running on GCE node
I20260925 16:42:11.884171 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:11.884756 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:11.885936 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790354531885907 us; error 48 us; skew 500 ppm
Sep 25 16:42:11 dist-test-slave-kxl8 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:11.888571 17482 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:42:11.889429 17482 webserver.cc:460] Webserver started at http://127.0.97.193:44105/ using document root <none> and password file <none>
I20260925 16:42:11.889978 17482 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:11.890066 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:11.890291 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:11.892345 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "c669dca4ddc6405cbb42dda0b95b428b"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "dbb12561332a9acad8744a3debfe9b7b"
server_key_iv: "6741c11e59bb6aaa337f515930a54b85"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.892916 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "c669dca4ddc6405cbb42dda0b95b428b"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "dbb12561332a9acad8744a3debfe9b7b"
server_key_iv: "6741c11e59bb6aaa337f515930a54b85"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.896461 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:42:11.898908 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:11.900031 17482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:11.900161 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "c669dca4ddc6405cbb42dda0b95b428b"
format_stamp: "Formatted at 2026-09-25 16:42:11 on dist-test-slave-kxl8"
server_key: "dbb12561332a9acad8744a3debfe9b7b"
server_key_iv: "6741c11e59bb6aaa337f515930a54b85"
server_key_version: "encryptionkey@0"
I20260925 16:42:11.900272 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:11.920154 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:11.923573 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:11.923779 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:11.924369 17482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:11.925200 17482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:11.925271 17482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:11.925380 17482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:11.925432 17482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:11.933805 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:39371
I20260925 16:42:11.933830 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:39371 every 8 connection(s)
I20260925 16:42:11.934653 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:11.942788 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17482
I20260925 16:42:11.942939 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:42:11.943259 391 external_mini_cluster.cc:1428] Setting key f19b0f4b1900b0e0f25e6017c1d4b151
Sep 25 16:42:11 dist-test-slave-kxl8 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:11.949921 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:11.936543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:56085 (local address 127.0.97.254:44523)
0925 16:42:11.936688 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:11.936693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:11.938386 (+ 1693us) server_negotiation.cc:407] Connection header received
0925 16:42:11.939623 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:11.939626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:11.939673 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:11.939751 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:11.940843 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.941481 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.942575 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.942806 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.942914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:11.945715 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:11.945718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:11.945720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:11.945742 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:11.947505 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:11.948170 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:11.948172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:11.948174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:11.948249 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:11.948635 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:11.948637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:11.948638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:11.948823 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:11.948911 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:11.949623 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 16:42:11.949755 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 16:42:11.951287 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:11.936982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44523 (local address 127.0.97.193:56085)
0925 16:42:11.938195 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:11.938252 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:11.939422 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:11.939913 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:11.939919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:11.940253 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:11.940714 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:11.940727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.941587 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.941590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:11.942450 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:11.942458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:11.942989 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:11.942997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:11.943195 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:11.943820 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:11.943856 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:11.945549 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:11.947664 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:11.947672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:11.947690 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:11.948066 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:11.948339 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:11.948345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:11.948350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:11.948542 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:11.948956 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:11.948966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:11.949343 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 16:42:11.949579 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 16:42:11.949967 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":157,"threads_started":1}
I20260925 16:42:11.952548 17611 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44523
I20260925 16:42:11.952859 17611 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:11.953537 17611 heartbeater.cc:511] Master 127.0.97.254:44523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:42:11.955260 17420 ts_manager.cc:194] Registered new tserver with Master: c669dca4ddc6405cbb42dda0b95b428b (127.0.97.193:39371)
I20260925 16:42:11.956449 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:56085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:42:11.973958 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:12.082455 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:12.082739 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:12.082813 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:12.086791 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:12.086874 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:12.086964 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:12.092284 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:12.093366 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:12.094393 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:12.103958 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:12.104027 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:12.103904 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.104318 17616 server_base.cc:1034] running on GCE node
I20260925 16:42:12.104923 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:12.105613 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:12.106775 17616 hybrid_clock.cc:648] HybridClock initialized: now 1790354532106759 us; error 28 us; skew 500 ppm
Sep 25 16:42:12 dist-test-slave-kxl8 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:12.109567 17616 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:42:12.110411 17616 webserver.cc:460] Webserver started at http://127.0.97.194:44483/ using document root <none> and password file <none>
I20260925 16:42:12.111042 17616 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:12.111133 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:12.111377 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:12.113435 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "a2078ec7ce194a8ab556bfd777818a8e"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "75f7f8888c7626dad6a6c077f7d3c34f"
server_key_iv: "cb5880091c0e3abb7d5bef04fccc9125"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.114008 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "a2078ec7ce194a8ab556bfd777818a8e"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "75f7f8888c7626dad6a6c077f7d3c34f"
server_key_iv: "cb5880091c0e3abb7d5bef04fccc9125"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.117604 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:12.119933 17631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.121079 17616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:12.121213 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "a2078ec7ce194a8ab556bfd777818a8e"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "75f7f8888c7626dad6a6c077f7d3c34f"
server_key_iv: "cb5880091c0e3abb7d5bef04fccc9125"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.121381 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:12.131623 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:12.134523 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:12.134719 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:12.135319 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:12.136137 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:12.136207 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.136279 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:12.136322 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.145154 17616 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:37041
I20260925 16:42:12.145223 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:37041 every 8 connection(s)
I20260925 16:42:12.146124 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:12.149878 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17616
I20260925 16:42:12.150030 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:12.150355 391 external_mini_cluster.cc:1428] Setting key 5fddd2a2a65c0cf0fc8cea5dddf9e965
Sep 25 16:42:12 dist-test-slave-kxl8 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:42:12.160426 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.147977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:35417 (local address 127.0.97.254:44523)
0925 16:42:12.148127 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.148131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.149016 (+ 885us) server_negotiation.cc:407] Connection header received
0925 16:42:12.150001 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.150004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.150059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.150153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:12.151640 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.152197 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.153500 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.153713 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.153798 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.156742 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:12.156750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:12.156753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:12.156775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:12.158366 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.158915 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.158919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.158920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.158996 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.159347 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.159352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.159354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.159532 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:12.159636 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.160151 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.160287 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:42:12.161180 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.148348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44523 (local address 127.0.97.194:35417)
0925 16:42:12.148841 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.148885 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.149786 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.150351 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.150361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.150842 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:12.151467 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.151486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.152344 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.152348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.153375 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.153389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.153877 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.153882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.154099 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.154753 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:12.154777 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:12.156557 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:12.158526 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:12.158532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:12.158547 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:12.158801 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:12.159087 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:12.159091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:12.159093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:12.159249 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:12.159634 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:12.159641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:12.159878 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.160082 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.160331 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":111,"threads_started":1}
I20260925 16:42:12.162353 17745 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44523
I20260925 16:42:12.162601 17745 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:42:12.163256 17745 heartbeater.cc:511] Master 127.0.97.254:44523 requested a full tablet report, sending...
I20260925 16:42:12.164346 17420 ts_manager.cc:194] Registered new tserver with Master: a2078ec7ce194a8ab556bfd777818a8e (127.0.97.194:37041)
I20260925 16:42:12.164925 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:35417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:42:12.180169 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:12.289232 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:12.289506 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:12.289546 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:12.293519 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:12.293572 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:12.293629 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:12.298655 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:12.299587 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:12.300514 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:12.309729 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:12.309733 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.309818 17750 server_base.cc:1034] running on GCE node
W20260925 16:42:12.309855 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.310357 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:12.311021 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:12.312197 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790354532312186 us; error 51 us; skew 500 ppm
Sep 25 16:42:12 dist-test-slave-kxl8 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:12.314926 17750 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:42:12.315784 17750 webserver.cc:460] Webserver started at http://127.0.97.195:41985/ using document root <none> and password file <none>
I20260925 16:42:12.316291 17750 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:12.316350 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:12.316519 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:12.318545 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "06610fd816124682aadbabd624a46eca"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "b89add90a2111153950b74226f82db49"
server_key_iv: "444811f827c57c78d405a5e1b6a2210d"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.319056 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "06610fd816124682aadbabd624a46eca"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "b89add90a2111153950b74226f82db49"
server_key_iv: "444811f827c57c78d405a5e1b6a2210d"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.322635 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:12.324975 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.326236 17750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:12.326354 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "06610fd816124682aadbabd624a46eca"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "b89add90a2111153950b74226f82db49"
server_key_iv: "444811f827c57c78d405a5e1b6a2210d"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.326460 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:12.344357 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:12.347234 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:12.347436 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:12.348045 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:12.348866 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:12.348920 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.348985 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:12.349018 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.357695 17750 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37641
I20260925 16:42:12.357765 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37641 every 8 connection(s)
I20260925 16:42:12.358613 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:12.366129 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17750
I20260925 16:42:12.366293 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:12.366623 391 external_mini_cluster.cc:1428] Setting key 92b0f7ba883b3b79bf215e0845a8f163
Sep 25 16:42:12 dist-test-slave-kxl8 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:12.371778 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.360384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:59045 (local address 127.0.97.254:44523)
0925 16:42:12.360524 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.360528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.361383 (+ 855us) server_negotiation.cc:407] Connection header received
0925 16:42:12.362319 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.362323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.362379 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.362495 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:12.363738 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.364265 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.365446 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.365665 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.365776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.368255 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:12.368259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:12.368262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:12.368287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:12.369778 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.370293 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.370298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.370299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.370378 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.370730 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.370735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.370736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.370914 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:12.371021 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.371511 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.371621 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:42:12.372546 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.360765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:44523 (local address 127.0.97.195:59045)
0925 16:42:12.361198 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.361240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.362114 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.362671 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.362678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.363023 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:12.363592 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.363606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.364390 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.364404 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.365316 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.365332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.365812 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.365816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.365985 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.366675 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:12.366691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:12.368112 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:12.369917 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:12.369922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:12.369935 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:12.370183 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:12.370492 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:12.370495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:12.370497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:12.370632 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:12.371028 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:12.371037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:12.371278 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.371467 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.371768 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260925 16:42:12.373629 17879 heartbeater.cc:347] Connected to a master server at 127.0.97.254:44523
I20260925 16:42:12.373870 17879 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:12.374382 17879 heartbeater.cc:511] Master 127.0.97.254:44523 requested a full tablet report, sending...
I20260925 16:42:12.375340 17420 ts_manager.cc:194] Registered new tserver with Master: 06610fd816124682aadbabd624a46eca (127.0.97.195:37641)
I20260925 16:42:12.375913 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:59045
I20260925 16:42:12.380388 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:12.388993 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.0.97.254:44523)
0925 16:42:12.382408 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.382412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.382623 (+ 211us) server_negotiation.cc:407] Connection header received
0925 16:42:12.382719 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.382722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.382755 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.382819 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:12.383460 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.383936 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.384703 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.384899 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.384972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.386237 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:12.386240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:12.386242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:12.386262 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:12.387688 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.388069 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.388072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.388073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.388142 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:12.388343 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:12.388344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:12.388345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:12.388537 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:12.388634 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.388804 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.388884 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:42:12.391659 17420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:12.394750 17420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:12.408393 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.402788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:37041 (local address 127.0.0.1:46352)
0925 16:42:12.403098 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.403144 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.403262 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.403850 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.403855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.403878 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:12.404029 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.404037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.405571 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.405574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.406717 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.406748 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.407756 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.407760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.407944 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.407993 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.408113 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.408239 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":29,"thread_start_us":87,"threads_started":1}
I20260925 16:42:12.408864 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.402930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.97.194:37041)
0925 16:42:12.403245 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.403264 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.403287 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:42:12.403415 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.403419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.403672 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.403848 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:12.404171 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.405426 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.406862 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.407627 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.407745 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.407901 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.408595 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.408700 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":136,"threads_started":1}
I20260925 16:42:12.409653 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.403497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37641 (local address 127.0.0.1:34568)
0925 16:42:12.404259 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.404274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.404363 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.405251 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.405254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.405292 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:12.405389 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.405395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.406771 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.406777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.407978 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.407988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.409227 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.409231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.409342 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.409365 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.409445 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.409515 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":39,"thread_start_us":116,"threads_started":1}
I20260925 16:42:12.409700 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.404422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.0.97.195:37641)
0925 16:42:12.404824 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.404841 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.404882 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:42:12.404960 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.404965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.405122 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.405276 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:12.405486 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.406586 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.408102 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.409105 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.409225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.409393 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.409475 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.409557 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":118,"threads_started":1}
I20260925 16:42:12.411962 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 (DEFAULT_TABLE table=test-table [id=d47e82b7eccb45a0a10475d70b19fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:12.412158 17813 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 (DEFAULT_TABLE table=test-table [id=d47e82b7eccb45a0a10475d70b19fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:12.412971 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ddd42040c4cfc9f16872065ddc649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:12.413070 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ddd42040c4cfc9f16872065ddc649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:12.413683 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.404389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.0.97.193:39371)
0925 16:42:12.404732 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.404747 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.404891 (+ 144us) server_negotiation.cc:407] Connection header received
0925 16:42:12.405057 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.405063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.405251 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.405408 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:12.405791 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.407218 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.408449 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.409333 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.409433 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.409562 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.413412 (+ 3850us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.413526 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260925 16:42:12.414049 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.403834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:39371 (local address 127.0.0.1:47152)
0925 16:42:12.404788 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.404802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.404908 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.405575 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.405577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.405594 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:12.405667 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.405673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.407355 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.407359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.408332 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.408339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.409621 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.409625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.409710 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.409737 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.413805 (+ 4068us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.413924 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":101,"threads_started":1}
I20260925 16:42:12.416910 17545 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 (DEFAULT_TABLE table=test-table [id=d47e82b7eccb45a0a10475d70b19fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:12.418021 17545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ddd42040c4cfc9f16872065ddc649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:12.419328 17896 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Bootstrap starting.
I20260925 16:42:12.420732 17897 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: Bootstrap starting.
I20260925 16:42:12.421629 17896 tablet_bootstrap.cc:654] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:12.422904 17897 tablet_bootstrap.cc:654] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:12.422899 17896 log.cc:824] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:12.424033 17897 log.cc:824] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:12.425040 17896 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: No bootstrap required, opened a new log
I20260925 16:42:12.425256 17896 ts_tablet_manager.cc:1397] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:12.425890 17900 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: Bootstrap starting.
I20260925 16:42:12.427811 17900 tablet_bootstrap.cc:654] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:12.427826 17897 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: No bootstrap required, opened a new log
I20260925 16:42:12.428025 17897 ts_tablet_manager.cc:1397] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:42:12.428975 17900 log.cc:824] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:12.429008 17896 raft_consensus.cc:348] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.429306 17896 raft_consensus.cc:374] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:12.429395 17896 raft_consensus.cc:729] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06610fd816124682aadbabd624a46eca, State: Initialized, Role: FOLLOWER
I20260925 16:42:12.429877 17896 consensus_queue.cc:260] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.430544 17879 heartbeater.cc:503] Master 127.0.97.254:44523 was elected leader, sending a full tablet report...
I20260925 16:42:12.430887 17896 ts_tablet_manager.cc:1428] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 16:42:12.431555 17900 tablet_bootstrap.cc:492] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: No bootstrap required, opened a new log
I20260925 16:42:12.431789 17900 ts_tablet_manager.cc:1397] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:12.432159 17897 raft_consensus.cc:348] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.432394 17897 raft_consensus.cc:374] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:12.432451 17897 raft_consensus.cc:729] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2078ec7ce194a8ab556bfd777818a8e, State: Initialized, Role: FOLLOWER
I20260925 16:42:12.432861 17897 consensus_queue.cc:260] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.433462 17745 heartbeater.cc:503] Master 127.0.97.254:44523 was elected leader, sending a full tablet report...
I20260925 16:42:12.434684 17897 ts_tablet_manager.cc:1428] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260925 16:42:12.435529 17900 raft_consensus.cc:348] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.435729 17900 raft_consensus.cc:374] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:12.435807 17900 raft_consensus.cc:729] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c669dca4ddc6405cbb42dda0b95b428b, State: Initialized, Role: FOLLOWER
I20260925 16:42:12.436215 17900 consensus_queue.cc:260] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.436791 17611 heartbeater.cc:503] Master 127.0.97.254:44523 was elected leader, sending a full tablet report...
I20260925 16:42:12.437106 17900 ts_tablet_manager.cc:1428] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:42:12.439122 17902 raft_consensus.cc:482] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:12.439286 17902 raft_consensus.cc:504] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.440385 17902 leader_election.cc:288] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2078ec7ce194a8ab556bfd777818a8e (127.0.97.194:37041), c669dca4ddc6405cbb42dda0b95b428b (127.0.97.193:39371)
I20260925 16:42:12.444068 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.440775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:57139 (local address 127.0.97.194:37041)
0925 16:42:12.440918 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.440923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.440938 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:42:12.440988 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.440991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.441039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.441121 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:12.441567 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.442075 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443119 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.443526 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.443703 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.443880 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.443950 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:42:12.444068 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.440659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:37041 (local address 127.0.97.195:57139)
0925 16:42:12.440805 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.440819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.440888 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.441249 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.441335 (+ 86us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.441354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:12.441457 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.441463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.442187 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.442192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.442999 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.443008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.443650 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.443762 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.443782 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.443866 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.443940 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":23}
I20260925 16:42:12.444463 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.441059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:39587 (local address 127.0.97.193:39371)
0925 16:42:12.441181 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.441185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.441384 (+ 199us) server_negotiation.cc:407] Connection header received
0925 16:42:12.441531 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.441534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.441577 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.441653 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:12.441917 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.442390 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443454 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.443841 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.444023 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.444255 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.444351 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:42:12.444453 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.440967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:39371 (local address 127.0.97.195:39587)
0925 16:42:12.441314 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.441327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.441433 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:12.441674 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:12.441677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:12.441692 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:12.441810 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.441815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.442499 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.442504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.443303 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:12.443312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.443967 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.443971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:12.444064 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.444085 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:42:12.444230 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 16:42:12.444306 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260925 16:42:12.444795 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ddd42040c4cfc9f16872065ddc649" candidate_uuid: "06610fd816124682aadbabd624a46eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2078ec7ce194a8ab556bfd777818a8e" is_pre_election: true
I20260925 16:42:12.445062 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ddd42040c4cfc9f16872065ddc649" candidate_uuid: "06610fd816124682aadbabd624a46eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c669dca4ddc6405cbb42dda0b95b428b" is_pre_election: true
I20260925 16:42:12.445142 17699 raft_consensus.cc:2454] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06610fd816124682aadbabd624a46eca in term 0.
I20260925 16:42:12.445358 17565 raft_consensus.cc:2454] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06610fd816124682aadbabd624a46eca in term 0.
I20260925 16:42:12.445695 17768 leader_election.cc:302] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06610fd816124682aadbabd624a46eca, a2078ec7ce194a8ab556bfd777818a8e; no voters:
I20260925 16:42:12.445935 17902 raft_consensus.cc:2781] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:12.446022 17902 raft_consensus.cc:482] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:12.446066 17902 raft_consensus.cc:3037] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:12.447055 17902 raft_consensus.cc:504] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.447369 17902 leader_election.cc:288] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [CANDIDATE]: Term 1 election: Requested vote from peers a2078ec7ce194a8ab556bfd777818a8e (127.0.97.194:37041), c669dca4ddc6405cbb42dda0b95b428b (127.0.97.193:39371)
I20260925 16:42:12.447664 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ddd42040c4cfc9f16872065ddc649" candidate_uuid: "06610fd816124682aadbabd624a46eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2078ec7ce194a8ab556bfd777818a8e"
I20260925 16:42:12.447726 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ddd42040c4cfc9f16872065ddc649" candidate_uuid: "06610fd816124682aadbabd624a46eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c669dca4ddc6405cbb42dda0b95b428b"
I20260925 16:42:12.447775 17699 raft_consensus.cc:3037] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:12.447873 17565 raft_consensus.cc:3037] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:12.448817 17699 raft_consensus.cc:2454] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06610fd816124682aadbabd624a46eca in term 1.
I20260925 16:42:12.449031 17565 raft_consensus.cc:2454] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06610fd816124682aadbabd624a46eca in term 1.
I20260925 16:42:12.449120 17768 leader_election.cc:302] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06610fd816124682aadbabd624a46eca, a2078ec7ce194a8ab556bfd777818a8e; no voters:
I20260925 16:42:12.449350 17902 raft_consensus.cc:2781] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:12.449595 17902 raft_consensus.cc:686] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 1 LEADER]: Becoming Leader. State: Replica: 06610fd816124682aadbabd624a46eca, State: Running, Role: LEADER
I20260925 16:42:12.449950 17902 consensus_queue.cc:237] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } }
I20260925 16:42:12.453117 17419 catalog_manager.cc:5495] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca reported cstate change: term changed from 0 to 1, leader changed from <none> to 06610fd816124682aadbabd624a46eca (127.0.97.195). New cstate: current_term: 1 leader_uuid: "06610fd816124682aadbabd624a46eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06610fd816124682aadbabd624a46eca" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 } health_report { overall_health: UNKNOWN } } }
I20260925 16:42:12.470863 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.467322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.0.97.195:37641)
0925 16:42:12.467476 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.467483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.467500 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:42:12.467584 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.467586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.467636 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.467744 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:12.467988 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.468493 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.469417 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.469641 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.469751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.470050 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:12.470454 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:12.470583 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.470645 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.470700 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:42:12.476559 17565 raft_consensus.cc:1261] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 1 FOLLOWER]: Refusing update from remote peer 06610fd816124682aadbabd624a46eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:12.476560 17699 raft_consensus.cc:1261] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 1 FOLLOWER]: Refusing update from remote peer 06610fd816124682aadbabd624a46eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:12.477336 17902 consensus_queue.cc:1035] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c669dca4ddc6405cbb42dda0b95b428b" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 39371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:12.477576 17906 consensus_queue.cc:1035] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2078ec7ce194a8ab556bfd777818a8e" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:12.485263 17911 mvcc.cc:205] Tried to move back new op lower bound from 7333292165016530944 to 7333292164917952512. Current Snapshot: MvccSnapshot[applied={T|T < 7333292165016530944}]
I20260925 16:42:12.496445 17420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:12.496611 17420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:12.499061 17420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b7417b7309149ee87fe263920216b6f: Sending DeleteTablet for 3 replicas of tablet 8d1ddd42040c4cfc9f16872065ddc649
I20260925 16:42:12.499888 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:12 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:46352
I20260925 16:42:12.500168 17545 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:12 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:47152
I20260925 16:42:12.500165 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ddd42040c4cfc9f16872065ddc649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:12 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:34568
I20260925 16:42:12.500631 17920 tablet_replica.cc:330] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: stopping tablet replica
I20260925 16:42:12.500629 17921 tablet_replica.cc:330] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b: stopping tablet replica
I20260925 16:42:12.500958 17920 raft_consensus.cc:2229] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:12.500958 17921 raft_consensus.cc:2229] T 8d1ddd42040c4cfc9f16872065ddc649 P c669dca4ddc6405cbb42dda0b95b428b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:12.501003 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17482
I20260925 16:42:12.502610 17920 raft_consensus.cc:2258] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:12.504437 17920 ts_tablet_manager.cc:1905] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:12.507761 17920 ts_tablet_manager.cc:1918] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:12.507946 17920 log.cc:1196] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/8d1ddd42040c4cfc9f16872065ddc649
I20260925 16:42:12.508361 17920 ts_tablet_manager.cc:1939] T 8d1ddd42040c4cfc9f16872065ddc649 P 06610fd816124682aadbabd624a46eca: Deleting consensus metadata
I20260925 16:42:12.509770 17406 catalog_manager.cc:4841] TS 06610fd816124682aadbabd624a46eca (127.0.97.195:37641): tablet 8d1ddd42040c4cfc9f16872065ddc649 (table test-table [id=d47e82b7eccb45a0a10475d70b19fb53]) successfully deleted
I20260925 16:42:12.510327 17919 tablet_replica.cc:330] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e: stopping tablet replica
I20260925 16:42:12.510700 17919 raft_consensus.cc:2229] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 16:42:12.510737 17406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:39371 (error 108)
I20260925 16:42:12.511096 17919 raft_consensus.cc:2258] T 8d1ddd42040c4cfc9f16872065ddc649 P a2078ec7ce194a8ab556bfd777818a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:12.511538 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17616
I20260925 16:42:12.512262 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.511519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:39371 (local address 127.0.0.1:47162)
0925 16:42:12.512102 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.512177 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493}
W20260925 16:42:12.513398 17406 catalog_manager.cc:4584] TS c669dca4ddc6405cbb42dda0b95b428b (127.0.97.193:39371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d1ddd42040c4cfc9f16872065ddc649: Network error: Client connection negotiation failed: client connection to 127.0.97.193:39371: connect: Connection refused (error 111)
I20260925 16:42:12.517699 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17750
I20260925 16:42:12.517904 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.517600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:37041 (local address 127.0.0.1:46356)
0925 16:42:12.517738 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.517799 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 16:42:12.523924 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17388
2026-09-25T16:42:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2256 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T16:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:12Z Disabled control of system clock
I20260925 16:42:12.553742 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34139
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:12.660961 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:12.661222 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:12.661258 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:12.665215 17926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:12.665306 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:12.665323 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:12.665339 17926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:12.665355 17926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:12.670140 17926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:12.671151 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:12.672466 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:12.680791 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:12.680852 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.680948 17926 server_base.cc:1034] running on GCE node
W20260925 16:42:12.680791 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.681720 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:12.682811 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:12.683966 17926 hybrid_clock.cc:648] HybridClock initialized: now 1790354532683937 us; error 44 us; skew 500 ppm
I20260925 16:42:12.685982 17926 webserver.cc:460] Webserver started at http://127.0.97.254:40535/ using document root <none> and password file <none>
I20260925 16:42:12.686566 17926 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:12.686632 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:12.686849 17926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:12.688884 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "469f80ad41dc40a8b950189597a24db5"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "e3b7ee103cc9ed41c78d6799e2d61fc2"
server_key_iv: "5234e79b23e9f1fba72fbfcf920e173f"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.689491 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "469f80ad41dc40a8b950189597a24db5"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "e3b7ee103cc9ed41c78d6799e2d61fc2"
server_key_iv: "5234e79b23e9f1fba72fbfcf920e173f"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.693151 17926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 16:42:12.695662 17940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.696913 17926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:12.697052 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "469f80ad41dc40a8b950189597a24db5"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "e3b7ee103cc9ed41c78d6799e2d61fc2"
server_key_iv: "5234e79b23e9f1fba72fbfcf920e173f"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.697168 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:12.736322 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:12.737093 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:12.737326 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:12.744669 17926 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:34139
I20260925 16:42:12.744691 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:34139 every 8 connection(s)
I20260925 16:42:12.745657 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:12.748744 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:12.749190 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 17926
I20260925 16:42:12.749346 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:12.749644 391 external_mini_cluster.cc:1428] Setting key c99dc43a16e3c76beda74db3c8fc35e8
I20260925 16:42:12.755023 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: Bootstrap starting.
I20260925 16:42:12.757737 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:12.758549 17993 log.cc:824] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:12.758898 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.751016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.97.254:34139)
0925 16:42:12.751639 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.751653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.751689 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:12.752329 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.752346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.752742 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.753029 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:12.753445 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.754390 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.755520 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:12.755974 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:12.756116 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:12.757065 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:12.757069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:12.757082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:12.757530 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:12.757585 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:12.757606 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:12.757680 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:12.757889 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 16:42:12.758124 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":154,"threads_started":1}
I20260925 16:42:12.760645 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: No bootstrap required, opened a new log
I20260925 16:42:12.764317 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } }
I20260925 16:42:12.764562 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:12.764598 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f80ad41dc40a8b950189597a24db5, State: Initialized, Role: FOLLOWER
I20260925 16:42:12.764966 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } }
I20260925 16:42:12.765082 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:12.765121 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:12.765208 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:12.766266 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } }
I20260925 16:42:12.766638 17993 leader_election.cc:302] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 469f80ad41dc40a8b950189597a24db5; no voters:
I20260925 16:42:12.767194 17993 leader_election.cc:288] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:12.767320 17998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:12.767616 17998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [term 1 LEADER]: Becoming Leader. State: Replica: 469f80ad41dc40a8b950189597a24db5, State: Running, Role: LEADER
I20260925 16:42:12.767989 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } }
I20260925 16:42:12.768616 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:12.769711 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 469f80ad41dc40a8b950189597a24db5. Latest consensus state: current_term: 1 leader_uuid: "469f80ad41dc40a8b950189597a24db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } } }
I20260925 16:42:12.769865 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:12.770141 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "469f80ad41dc40a8b950189597a24db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469f80ad41dc40a8b950189597a24db5" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34139 } } }
I20260925 16:42:12.770236 18007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:12.770288 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:12.773187 18007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:12.779147 18007 catalog_manager.cc:1347] Generated new cluster ID: c477abd1363841ed93515d726d495048
I20260925 16:42:12.779315 18007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:12.786207 18007 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:12.787521 18007 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:12.801595 18007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: Generated new TSK 0
I20260925 16:42:12.802379 18007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:12.808466 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--builtin_ntp_servers=127.0.97.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:12.931850 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:12.932089 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:12.932126 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:12.936141 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:12.936197 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:12.936254 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:12.941762 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:12.943100 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:12.944533 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:12.954103 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:12.954115 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.954362 18017 server_base.cc:1034] running on GCE node
W20260925 16:42:12.954128 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:12.954875 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:12.955555 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:12.956735 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790354532956711 us; error 37 us; skew 500 ppm
I20260925 16:42:12.958585 18017 webserver.cc:460] Webserver started at http://127.0.97.193:40809/ using document root <none> and password file <none>
I20260925 16:42:12.959164 18017 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:12.959259 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:12.959504 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:12.961642 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "4281c2ac6b154716b8c67e27238f91eb"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "7c229538f64519d2313a92a842bff054"
server_key_iv: "9b52332bc5e451e48325d04ee082a0d5"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.962199 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "4281c2ac6b154716b8c67e27238f91eb"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "7c229538f64519d2313a92a842bff054"
server_key_iv: "9b52332bc5e451e48325d04ee082a0d5"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.965826 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:12.968147 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.969374 18017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:12.969512 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "4281c2ac6b154716b8c67e27238f91eb"
format_stamp: "Formatted at 2026-09-25 16:42:12 on dist-test-slave-kxl8"
server_key: "7c229538f64519d2313a92a842bff054"
server_key_iv: "9b52332bc5e451e48325d04ee082a0d5"
server_key_version: "encryptionkey@0"
I20260925 16:42:12.969624 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:12.982878 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:12.983559 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:12.983760 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:12.984370 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:12.985423 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:12.985493 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.985556 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:12.985608 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:12.994355 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:42931
I20260925 16:42:12.994411 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:42931 every 8 connection(s)
I20260925 16:42:12.995296 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:13.004536 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.997119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:60361 (local address 127.0.97.254:34139)
0925 16:42:12.997265 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:12.997270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:12.998365 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 16:42:12.999709 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:12.999712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:12.999795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:12.999884 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:13.000887 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.001608 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.002742 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.002972 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.003065 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.004095 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:13.004098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:13.004100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:13.004112 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:13.004131 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:13.004140 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:13.004201 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.004332 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.004418 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:42:13.005400 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:12.997576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34139 (local address 127.0.97.193:60361)
0925 16:42:12.998207 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:12.998248 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:12.999482 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.000060 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.000067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.000193 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:13.000740 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.000753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.001726 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.001729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.002613 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.002622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.003113 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.003115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.003253 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.003847 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:13.003849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:13.003887 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:13.003955 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:13.004214 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:13.004220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:13.004224 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.004316 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.004527 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260925 16:42:13.005548 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18017
I20260925 16:42:13.005683 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:42:13.006017 391 external_mini_cluster.cc:1428] Setting key 5608bf12dc6f33f81b10b8826895da7e
I20260925 16:42:13.006776 18145 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34139
I20260925 16:42:13.007028 18145 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:13.007699 18145 heartbeater.cc:511] Master 127.0.97.254:34139 requested a full tablet report, sending...
I20260925 16:42:13.008074 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--builtin_ntp_servers=127.0.97.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:13.009276 17957 ts_manager.cc:194] Registered new tserver with Master: 4281c2ac6b154716b8c67e27238f91eb (127.0.97.193:42931)
I20260925 16:42:13.010815 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.193:60361
W20260925 16:42:13.116942 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:13.117200 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:13.117238 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:13.121205 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:13.121259 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:13.121367 18148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:13.126117 18148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:13.127202 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:13.128561 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:13.137969 18153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:13.138010 18154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:13.137980 18148 server_base.cc:1034] running on GCE node
W20260925 16:42:13.138000 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:13.138597 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:13.139269 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:13.140435 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790354533140414 us; error 33 us; skew 500 ppm
I20260925 16:42:13.142249 18148 webserver.cc:460] Webserver started at http://127.0.97.194:33461/ using document root <none> and password file <none>
I20260925 16:42:13.142824 18148 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:13.142885 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:13.143126 18148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:13.145346 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "135dcc3641a04d2695f1bfe8974a9018"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "b0334dfd4e588a28f34b0ef4db0f3df8"
server_key_iv: "10dc29717c16d47abf25d53db826476e"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.145968 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "135dcc3641a04d2695f1bfe8974a9018"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "b0334dfd4e588a28f34b0ef4db0f3df8"
server_key_iv: "10dc29717c16d47abf25d53db826476e"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.149608 18148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:42:13.152294 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.153553 18148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:13.153682 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "135dcc3641a04d2695f1bfe8974a9018"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "b0334dfd4e588a28f34b0ef4db0f3df8"
server_key_iv: "10dc29717c16d47abf25d53db826476e"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.153797 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:13.171176 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:13.171861 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:13.172066 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:13.172688 18148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:13.173738 18148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:13.173806 18148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.173875 18148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:13.173933 18148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.182248 18148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:44675
I20260925 16:42:13.182273 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:44675 every 8 connection(s)
I20260925 16:42:13.183172 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:13.184350 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18148
I20260925 16:42:13.184458 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:13.184765 391 external_mini_cluster.cc:1428] Setting key 9a1967d76472a002d96124def12517d2
I20260925 16:42:13.186761 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--builtin_ntp_servers=127.0.97.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:13.193943 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.185184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:51355 (local address 127.0.97.254:34139)
0925 16:42:13.185387 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.185391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.186531 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 16:42:13.188400 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.188403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.188458 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.188546 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:13.189497 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.190393 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.191614 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.192067 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.192195 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.193330 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:13.193337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:13.193343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:13.193361 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:13.193387 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:13.193401 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:13.193488 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.193639 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.193716 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:42:13.194687 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.185659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34139 (local address 127.0.97.194:51355)
0925 16:42:13.186353 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.186421 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.188173 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.188732 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.188739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.188823 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:13.189350 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.189366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.190519 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.190522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.191481 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.191491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.192226 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.192229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.192397 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.193036 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:13.193038 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:13.193079 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:13.193140 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:13.193493 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:13.193500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:13.193505 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.193604 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.193827 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260925 16:42:13.195889 18276 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34139
I20260925 16:42:13.196166 18276 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:13.196848 18276 heartbeater.cc:511] Master 127.0.97.254:34139 requested a full tablet report, sending...
I20260925 16:42:13.198057 17957 ts_manager.cc:194] Registered new tserver with Master: 135dcc3641a04d2695f1bfe8974a9018 (127.0.97.194:44675)
I20260925 16:42:13.198663 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.194:51355
W20260925 16:42:13.300489 18279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:13.300750 18279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:13.300812 18279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:13.305254 18279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:13.305379 18279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:13.305474 18279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:13.310325 18279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:13.311386 18279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:13.312805 18279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:13.322244 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:13.322257 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:13.322252 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:13.322521 18279 server_base.cc:1034] running on GCE node
I20260925 16:42:13.323016 18279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:13.323650 18279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:13.324851 18279 hybrid_clock.cc:648] HybridClock initialized: now 1790354533324822 us; error 41 us; skew 500 ppm
I20260925 16:42:13.326675 18279 webserver.cc:460] Webserver started at http://127.0.97.195:40741/ using document root <none> and password file <none>
I20260925 16:42:13.327257 18279 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:13.327356 18279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:13.327582 18279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:13.329623 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "4d2b3cc8aaad4ea1833f64a1afc53998"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "e24e4062a61218007af32d813dab2c7c"
server_key_iv: "47b8ed7291ac81f04b95d0f6187463cd"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.330168 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "4d2b3cc8aaad4ea1833f64a1afc53998"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "e24e4062a61218007af32d813dab2c7c"
server_key_iv: "47b8ed7291ac81f04b95d0f6187463cd"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.333765 18279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:13.336134 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.337462 18279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:13.337576 18279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "4d2b3cc8aaad4ea1833f64a1afc53998"
format_stamp: "Formatted at 2026-09-25 16:42:13 on dist-test-slave-kxl8"
server_key: "e24e4062a61218007af32d813dab2c7c"
server_key_iv: "47b8ed7291ac81f04b95d0f6187463cd"
server_key_version: "encryptionkey@0"
I20260925 16:42:13.337690 18279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:13.346807 18279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:13.347424 18279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:13.347621 18279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:13.348170 18279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:13.349099 18279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:13.349170 18279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.349234 18279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:13.349349 18279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:13.358194 18279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:39533
I20260925 16:42:13.358258 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:39533 every 8 connection(s)
I20260925 16:42:13.359167 18279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:13.363461 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18279
I20260925 16:42:13.363546 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:13.363847 391 external_mini_cluster.cc:1428] Setting key c8646a488c38322a50d907ab17810656
I20260925 16:42:13.368659 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.360964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:48067 (local address 127.0.97.254:34139)
0925 16:42:13.361114 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.361118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.362370 (+ 1252us) server_negotiation.cc:407] Connection header received
0925 16:42:13.363794 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.363798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.363861 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.363967 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:13.365179 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.365824 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.366980 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.367194 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.367272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.368250 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:13.368253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:13.368255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:13.368266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:13.368285 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:13.368292 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:13.368349 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.368502 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.368550 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:42:13.369630 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.361491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34139 (local address 127.0.97.195:48067)
0925 16:42:13.362156 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.362196 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.363565 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.364177 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.364188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.364361 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:13.365040 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.365059 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.365952 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.365956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.366881 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.366890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.367317 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.367320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.367477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.368023 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:13.368025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:13.368063 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:13.368122 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:13.368385 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:13.368391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:13.368395 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.368488 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.368694 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260925 16:42:13.370744 18407 heartbeater.cc:347] Connected to a master server at 127.0.97.254:34139
I20260925 16:42:13.370939 18407 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:13.371428 18407 heartbeater.cc:511] Master 127.0.97.254:34139 requested a full tablet report, sending...
I20260925 16:42:13.372344 17957 ts_manager.cc:194] Registered new tserver with Master: 4d2b3cc8aaad4ea1833f64a1afc53998 (127.0.97.195:39533)
I20260925 16:42:13.373015 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.195:48067
I20260925 16:42:13.377622 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:13.382826 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.379536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.97.254:34139)
0925 16:42:13.379697 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.379701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.379785 (+ 84us) server_negotiation.cc:407] Connection header received
0925 16:42:13.379866 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.379868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.379903 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.379964 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:13.380208 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.380680 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.381450 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.381652 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.381722 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.382416 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:13.382419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:13.382421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:13.382431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:13.382448 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:13.382456 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:13.382513 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.382597 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.382642 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:42:13.385329 17957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:13.388232 17957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:13.402905 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.397657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42931 (local address 127.0.0.1:43444)
0925 16:42:13.397985 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.397998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.398077 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.398533 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.398536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.398551 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:13.398629 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.398633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.399799 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.399805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.401410 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.401421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.402372 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.402376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.402486 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.402531 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.402635 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.402750 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":147,"thread_start_us":91,"threads_started":1}
I20260925 16:42:13.403115 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.397621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.0.97.195:39533)
0925 16:42:13.398798 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.398808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.398830 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:42:13.398899 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.398916 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.399083 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.399229 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:13.399472 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.400451 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.401906 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.402504 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.402625 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.402805 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.402854 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.402950 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":100,"threads_started":1}
I20260925 16:42:13.403692 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.397670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.0.97.193:42931)
0925 16:42:13.398102 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.398114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.398138 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:42:13.398207 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.398212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.398401 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.398535 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:13.398779 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.399680 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.401546 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.402232 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.402317 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.402462 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.403414 (+ 952us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.403512 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":120,"threads_started":1}
I20260925 16:42:13.403851 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.396634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:44675 (local address 127.0.0.1:52884)
0925 16:42:13.397193 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.397242 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.397403 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.398142 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.398149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.398173 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:13.398307 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.398314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.399896 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.399899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.401410 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.401421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.403428 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.403432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.403516 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.403538 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.403620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.403681 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":77,"spinlock_wait_cycles":3328,"thread_start_us":59,"threads_started":1}
I20260925 16:42:13.403070 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.396828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39533 (local address 127.0.0.1:33766)
0925 16:42:13.397208 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.397261 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.397401 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.399228 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.399231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.399258 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:13.399350 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.399355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.400579 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.400586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.401788 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.401799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.402630 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.402633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.402719 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.402742 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.402804 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.402865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"spinlock_wait_cycles":5760,"thread_start_us":105,"threads_started":1}
I20260925 16:42:13.404256 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.396819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.97.194:44675)
0925 16:42:13.397621 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.397638 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.397662 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:42:13.397736 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.397743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.397962 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.398476 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:13.398616 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.399741 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.402515 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.403306 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.403389 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.403514 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.403662 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.404085 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":134,"threads_started":1}
I20260925 16:42:13.406986 18079 tablet_service.cc:1467] Processing CreateTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 (DEFAULT_TABLE table=test-table [id=cb6c715111e943cd8b15a556f55c5347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:13.407025 18210 tablet_service.cc:1467] Processing CreateTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 (DEFAULT_TABLE table=test-table [id=cb6c715111e943cd8b15a556f55c5347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:13.407025 18341 tablet_service.cc:1467] Processing CreateTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 (DEFAULT_TABLE table=test-table [id=cb6c715111e943cd8b15a556f55c5347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:13.408025 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a2c30ee3b438c9056c808cc2bd371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:13.408023 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a2c30ee3b438c9056c808cc2bd371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:13.408034 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a2c30ee3b438c9056c808cc2bd371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:13.414251 18424 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Bootstrap starting.
I20260925 16:42:13.414909 18426 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Bootstrap starting.
I20260925 16:42:13.415596 18425 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Bootstrap starting.
I20260925 16:42:13.417218 18425 tablet_bootstrap.cc:654] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:13.417237 18424 tablet_bootstrap.cc:654] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:13.417237 18426 tablet_bootstrap.cc:654] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:13.418108 18424 log.cc:824] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:13.418108 18426 log.cc:824] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:13.418108 18425 log.cc:824] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:13.419871 18424 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: No bootstrap required, opened a new log
I20260925 16:42:13.420102 18425 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: No bootstrap required, opened a new log
I20260925 16:42:13.420101 18424 ts_tablet_manager.cc:1397] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:42:13.420289 18425 ts_tablet_manager.cc:1397] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:42:13.421152 18426 tablet_bootstrap.cc:492] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: No bootstrap required, opened a new log
I20260925 16:42:13.421355 18426 ts_tablet_manager.cc:1397] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:42:13.423446 18425 raft_consensus.cc:348] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.423681 18425 raft_consensus.cc:374] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:13.423753 18425 raft_consensus.cc:729] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135dcc3641a04d2695f1bfe8974a9018, State: Initialized, Role: FOLLOWER
I20260925 16:42:13.424206 18425 consensus_queue.cc:260] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.424455 18424 raft_consensus.cc:348] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.424731 18424 raft_consensus.cc:374] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:13.424800 18424 raft_consensus.cc:729] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4281c2ac6b154716b8c67e27238f91eb, State: Initialized, Role: FOLLOWER
I20260925 16:42:13.425267 18425 ts_tablet_manager.cc:1428] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:42:13.425290 18424 consensus_queue.cc:260] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.425433 18426 raft_consensus.cc:348] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.425671 18276 heartbeater.cc:503] Master 127.0.97.254:34139 was elected leader, sending a full tablet report...
I20260925 16:42:13.425733 18426 raft_consensus.cc:374] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:13.425784 18426 raft_consensus.cc:729] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2b3cc8aaad4ea1833f64a1afc53998, State: Initialized, Role: FOLLOWER
I20260925 16:42:13.426157 18426 consensus_queue.cc:260] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.426285 18424 ts_tablet_manager.cc:1428] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:13.426585 18145 heartbeater.cc:503] Master 127.0.97.254:34139 was elected leader, sending a full tablet report...
I20260925 16:42:13.426780 18407 heartbeater.cc:503] Master 127.0.97.254:34139 was elected leader, sending a full tablet report...
I20260925 16:42:13.427057 18426 ts_tablet_manager.cc:1428] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 16:42:13.434046 18277 tablet.cc:2369] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:42:13.496474 18146 tablet.cc:2369] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:42:13.610173 18408 tablet.cc:2369] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:13.626334 18430 raft_consensus.cc:482] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:13.626600 18430 raft_consensus.cc:504] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.627936 18430 leader_election.cc:288] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d2b3cc8aaad4ea1833f64a1afc53998 (127.0.97.195:39533), 4281c2ac6b154716b8c67e27238f91eb (127.0.97.193:42931)
I20260925 16:42:13.631896 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.628126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:39533 (local address 127.0.97.194:55835)
0925 16:42:13.628277 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.628294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.628387 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.628710 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.628716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.628738 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:13.628878 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.628888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.629799 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.629802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.630723 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.630732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.631472 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.631476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.631575 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.631601 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.631687 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.631756 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":33}
I20260925 16:42:13.631949 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.628301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:55835 (local address 127.0.97.195:39533)
0925 16:42:13.628454 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.628460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.628474 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:42:13.628537 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.628539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.628597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.628672 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:13.629021 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.629690 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.630819 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.631355 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.631448 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.631484 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.631727 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.631801 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:42:13.632158 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.628416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42931 (local address 127.0.97.194:39317)
0925 16:42:13.628786 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.628808 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:13.628884 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:13.629115 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:13.629118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:13.629135 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:13.629252 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.629258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.630027 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.630029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.631010 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:13.631020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.631757 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.631761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:13.631851 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.631873 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:42:13.631951 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:42:13.632019 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":18,"thread_start_us":136,"threads_started":1}
I20260925 16:42:13.632197 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.628556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:39317 (local address 127.0.97.193:42931)
0925 16:42:13.628695 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.628699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.628892 (+ 193us) server_negotiation.cc:407] Connection header received
0925 16:42:13.628955 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.628961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.629017 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.629081 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:13.629389 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.629922 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.631138 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.631624 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.631731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.631770 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.631992 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.632074 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:42:13.632962 18099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a2c30ee3b438c9056c808cc2bd371" candidate_uuid: "135dcc3641a04d2695f1bfe8974a9018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4281c2ac6b154716b8c67e27238f91eb" is_pre_election: true
I20260925 16:42:13.632968 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a2c30ee3b438c9056c808cc2bd371" candidate_uuid: "135dcc3641a04d2695f1bfe8974a9018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" is_pre_election: true
I20260925 16:42:13.633296 18361 raft_consensus.cc:2454] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135dcc3641a04d2695f1bfe8974a9018 in term 0.
I20260925 16:42:13.633294 18099 raft_consensus.cc:2454] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135dcc3641a04d2695f1bfe8974a9018 in term 0.
I20260925 16:42:13.633751 18164 leader_election.cc:302] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135dcc3641a04d2695f1bfe8974a9018, 4281c2ac6b154716b8c67e27238f91eb; no voters:
I20260925 16:42:13.633975 18430 raft_consensus.cc:2781] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:13.634071 18430 raft_consensus.cc:482] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:13.634128 18430 raft_consensus.cc:3037] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:13.635113 18430 raft_consensus.cc:504] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.635444 18430 leader_election.cc:288] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [CANDIDATE]: Term 1 election: Requested vote from peers 4d2b3cc8aaad4ea1833f64a1afc53998 (127.0.97.195:39533), 4281c2ac6b154716b8c67e27238f91eb (127.0.97.193:42931)
I20260925 16:42:13.635764 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a2c30ee3b438c9056c808cc2bd371" candidate_uuid: "135dcc3641a04d2695f1bfe8974a9018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998"
I20260925 16:42:13.635842 18099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a2c30ee3b438c9056c808cc2bd371" candidate_uuid: "135dcc3641a04d2695f1bfe8974a9018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4281c2ac6b154716b8c67e27238f91eb"
I20260925 16:42:13.635921 18361 raft_consensus.cc:3037] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:13.635978 18099 raft_consensus.cc:3037] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:13.636911 18361 raft_consensus.cc:2454] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135dcc3641a04d2695f1bfe8974a9018 in term 1.
I20260925 16:42:13.637051 18099 raft_consensus.cc:2454] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135dcc3641a04d2695f1bfe8974a9018 in term 1.
I20260925 16:42:13.637212 18164 leader_election.cc:302] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135dcc3641a04d2695f1bfe8974a9018, 4d2b3cc8aaad4ea1833f64a1afc53998; no voters:
I20260925 16:42:13.637392 18430 raft_consensus.cc:2781] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:13.637600 18430 raft_consensus.cc:686] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 1 LEADER]: Becoming Leader. State: Replica: 135dcc3641a04d2695f1bfe8974a9018, State: Running, Role: LEADER
I20260925 16:42:13.637919 18430 consensus_queue.cc:237] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } }
I20260925 16:42:13.641103 17956 catalog_manager.cc:5495] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 135dcc3641a04d2695f1bfe8974a9018 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "135dcc3641a04d2695f1bfe8974a9018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135dcc3641a04d2695f1bfe8974a9018" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 44675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 } health_report { overall_health: UNKNOWN } } }
I20260925 16:42:13.692660 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.689477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.0.97.194:44675)
0925 16:42:13.689611 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:13.689615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:13.689628 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:42:13.689773 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:13.689775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:13.689836 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:13.689936 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:13.690125 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.690612 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.691402 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:13.691603 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:13.691677 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:13.691919 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:13.692336 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:13.692429 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:13.692505 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 16:42:13.692551 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:42:13.698464 18361 raft_consensus.cc:1261] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 1 FOLLOWER]: Refusing update from remote peer 135dcc3641a04d2695f1bfe8974a9018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:13.698465 18099 raft_consensus.cc:1261] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb [term 1 FOLLOWER]: Refusing update from remote peer 135dcc3641a04d2695f1bfe8974a9018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:13.699162 18430 consensus_queue.cc:1035] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d2b3cc8aaad4ea1833f64a1afc53998" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:13.699412 18434 consensus_queue.cc:1035] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4281c2ac6b154716b8c67e27238f91eb" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:13.706319 18437 mvcc.cc:205] Tried to move back new op lower bound from 7333292170021666816 to 7333292169783136256. Current Snapshot: MvccSnapshot[applied={T|T < 7333292170021666816}]
I20260925 16:42:13.708529 18438 mvcc.cc:205] Tried to move back new op lower bound from 7333292170021666816 to 7333292169783136256. Current Snapshot: MvccSnapshot[applied={T|T < 7333292170021666816}]
I20260925 16:42:13.713994 17956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:13.714178 17956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:13.716571 17956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 469f80ad41dc40a8b950189597a24db5: Sending DeleteTablet for 3 replicas of tablet 199a2c30ee3b438c9056c808cc2bd371
I20260925 16:42:13.717448 18079 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:13 UTC) from {username='slave'} at 127.0.0.1:43444
I20260925 16:42:13.717593 18341 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:13 UTC) from {username='slave'} at 127.0.0.1:33766
I20260925 16:42:13.717870 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a2c30ee3b438c9056c808cc2bd371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:13 UTC) from {username='slave'} at 127.0.0.1:52884
I20260925 16:42:13.718075 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18017
I20260925 16:42:13.718222 18446 tablet_replica.cc:330] T 199a2c30ee3b438c9056c808cc2bd371 P 4281c2ac6b154716b8c67e27238f91eb: stopping tablet replica
I20260925 16:42:13.719053 18447 tablet_replica.cc:330] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: stopping tablet replica
I20260925 16:42:13.719395 18447 raft_consensus.cc:2229] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:13.719723 18447 raft_consensus.cc:2258] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:13.720114 18448 tablet_replica.cc:330] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: stopping tablet replica
I20260925 16:42:13.720467 18448 raft_consensus.cc:2229] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:13.720884 18448 raft_consensus.cc:2258] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:13.721480 18447 ts_tablet_manager.cc:1905] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:13.722601 18448 ts_tablet_manager.cc:1905] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:13.724310 18447 ts_tablet_manager.cc:1918] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:13.724469 18447 log.cc:1196] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/199a2c30ee3b438c9056c808cc2bd371
I20260925 16:42:13.724846 18447 ts_tablet_manager.cc:1939] T 199a2c30ee3b438c9056c808cc2bd371 P 4d2b3cc8aaad4ea1833f64a1afc53998: Deleting consensus metadata
I20260925 16:42:13.726264 18448 ts_tablet_manager.cc:1918] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:13.726243 17942 catalog_manager.cc:4841] TS 4d2b3cc8aaad4ea1833f64a1afc53998 (127.0.97.195:39533): tablet 199a2c30ee3b438c9056c808cc2bd371 (table test-table [id=cb6c715111e943cd8b15a556f55c5347]) successfully deleted
I20260925 16:42:13.726444 18448 log.cc:1196] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/199a2c30ee3b438c9056c808cc2bd371
I20260925 16:42:13.726779 18448 ts_tablet_manager.cc:1939] T 199a2c30ee3b438c9056c808cc2bd371 P 135dcc3641a04d2695f1bfe8974a9018: Deleting consensus metadata
I20260925 16:42:13.727926 17944 catalog_manager.cc:4841] TS 135dcc3641a04d2695f1bfe8974a9018 (127.0.97.194:44675): tablet 199a2c30ee3b438c9056c808cc2bd371 (table test-table [id=cb6c715111e943cd8b15a556f55c5347]) successfully deleted
I20260925 16:42:13.728307 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18148
W20260925 16:42:13.728384 17942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:42931 (error 108)
I20260925 16:42:13.729936 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:13.729599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42931 (local address 127.0.0.1:43448)
0925 16:42:13.729754 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:13.729833 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:42931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:42:13.730156 17942 catalog_manager.cc:4584] TS 4281c2ac6b154716b8c67e27238f91eb (127.0.97.193:42931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 199a2c30ee3b438c9056c808cc2bd371: Network error: Client connection negotiation failed: client connection to 127.0.97.193:42931: connect: Connection refused (error 111)
I20260925 16:42:13.733688 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18279
I20260925 16:42:13.742434 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 17926
2026-09-25T16:42:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:13 dist-test-slave-kxl8 krb5kdc[18452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:13 dist-test-slave-kxl8 krb5kdc[18452](info): set up 2 sockets
Sep 25 16:42:13 dist-test-slave-kxl8 krb5kdc[18452](info): commencing operation
krb5kdc: starting...
W20260925 16:42:14.793226 391 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:14 dist-test-slave-kxl8 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:14.913126 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45103
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:45103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:15.024137 18465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:15.024408 18465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:15.024477 18465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:15.028616 18465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:15.028669 18465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:15.028688 18465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:15.028725 18465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:15.028759 18465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:15.034247 18465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:45103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:15.035276 18465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:15.036314 18465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:15.044765 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:15.044826 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.044945 18465 server_base.cc:1034] running on GCE node
W20260925 16:42:15.044891 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.045682 18465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:15.046811 18465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:15.048010 18465 hybrid_clock.cc:648] HybridClock initialized: now 1790354535047991 us; error 32 us; skew 500 ppm
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:15.051115 18465 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:15.052009 18465 webserver.cc:460] Webserver started at http://127.0.97.254:44035/ using document root <none> and password file <none>
I20260925 16:42:15.052567 18465 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:15.052660 18465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:15.052896 18465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:15.054958 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "7d7298edb3464490873052fbaeadbacd"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "d68fe434708dacb16ae65f2b404687b7"
server_key_iv: "bc0d66084edd3b183d9e3c7d569c4c86"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.055524 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "7d7298edb3464490873052fbaeadbacd"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "d68fe434708dacb16ae65f2b404687b7"
server_key_iv: "bc0d66084edd3b183d9e3c7d569c4c86"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.059209 18465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 16:42:15.061731 18480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.062992 18465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:15.063138 18465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "7d7298edb3464490873052fbaeadbacd"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "d68fe434708dacb16ae65f2b404687b7"
server_key_iv: "bc0d66084edd3b183d9e3c7d569c4c86"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.063251 18465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:15.076663 18465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:15.079483 18465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:15.079684 18465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:15.086506 18465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:45103
I20260925 16:42:15.086534 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:45103 every 8 connection(s)
I20260925 16:42:15.087420 18465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:15.089129 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18465
I20260925 16:42:15.089272 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:15.089583 391 external_mini_cluster.cc:1428] Setting key fca5ce1e5aa7869b40cc75016a6cad9d
I20260925 16:42:15.090236 18533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:15.095870 18533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: Bootstrap starting.
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:15.098378 18533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:15.099123 18533 log.cc:824] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:15.100996 18533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: No bootstrap required, opened a new log
I20260925 16:42:15.103258 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.090974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.97.254:45103)
0925 16:42:15.091444 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.091459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.091495 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:15.092123 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.092138 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.092452 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.092782 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:15.093594 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.094589 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.095461 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.095936 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.096113 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.098757 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:15.098761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:15.098774 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:15.098800 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:15.100522 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.100976 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.100981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.100985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.101164 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.101481 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.101486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.101488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.101863 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:15.102064 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.102335 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.102611 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260925 16:42:15.104619 18533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } }
I20260925 16:42:15.104866 18533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:15.104920 18533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7298edb3464490873052fbaeadbacd, State: Initialized, Role: FOLLOWER
I20260925 16:42:15.105399 18533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } }
I20260925 16:42:15.105572 18533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:15.105626 18533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:15.105695 18533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:15.106470 18533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } }
I20260925 16:42:15.106737 18533 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7298edb3464490873052fbaeadbacd; no voters:
I20260925 16:42:15.106962 18533 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:15.107076 18538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:15.107332 18538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [term 1 LEADER]: Becoming Leader. State: Replica: 7d7298edb3464490873052fbaeadbacd, State: Running, Role: LEADER
I20260925 16:42:15.107726 18538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } }
I20260925 16:42:15.108103 18533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:15.109395 18539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7298edb3464490873052fbaeadbacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } } }
I20260925 16:42:15.109373 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7298edb3464490873052fbaeadbacd. Latest consensus state: current_term: 1 leader_uuid: "7d7298edb3464490873052fbaeadbacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7298edb3464490873052fbaeadbacd" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 45103 } } }
I20260925 16:42:15.109579 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:15.109579 18539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd [sys.catalog]: This master's current role is: LEADER
W20260925 16:42:15.112478 18553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:15.112573 18553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:15.112643 18549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:15.113351 18549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:15.118494 18549 catalog_manager.cc:1347] Generated new cluster ID: 8c5691bae5c84584bb89b30b20646ab3
I20260925 16:42:15.118556 18549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:15.128877 18549 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:15.129809 18549 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:15.137055 18549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: Generated new TSK 0
I20260925 16:42:15.137805 18549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:42:15.166612 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:15.277617 18559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:15.277874 18559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:15.277917 18559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:15.281754 18559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:15.281802 18559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:15.281859 18559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:15.286825 18559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:15.287765 18559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:15.288713 18559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:15.298295 18567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:15.298280 18565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:15.298276 18564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.298564 18559 server_base.cc:1034] running on GCE node
I20260925 16:42:15.298990 18559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:15.299635 18559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:15.300796 18559 hybrid_clock.cc:648] HybridClock initialized: now 1790354535300778 us; error 34 us; skew 500 ppm
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:15.303614 18559 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:42:15.304497 18559 webserver.cc:460] Webserver started at http://127.0.97.193:42767/ using document root <none> and password file <none>
I20260925 16:42:15.305050 18559 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:15.305140 18559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:15.305437 18559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:15.307441 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "a2b20dbb80214594aef47d49d16a2270"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "e67824d2e9a6180bd84f9382cc0029ce"
server_key_iv: "0b3a3e9cfef8e332c5600cb9ec462382"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.308048 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "a2b20dbb80214594aef47d49d16a2270"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "e67824d2e9a6180bd84f9382cc0029ce"
server_key_iv: "0b3a3e9cfef8e332c5600cb9ec462382"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.311949 18559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:42:15.314427 18574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.315661 18559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:15.315800 18559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "a2b20dbb80214594aef47d49d16a2270"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "e67824d2e9a6180bd84f9382cc0029ce"
server_key_iv: "0b3a3e9cfef8e332c5600cb9ec462382"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.315912 18559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:15.348117 18559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:15.351366 18559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:15.351577 18559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:15.352171 18559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:15.353062 18559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:15.353133 18559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.353205 18559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:15.353253 18559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.362049 18559 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:36105
I20260925 16:42:15.362075 18687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:36105 every 8 connection(s)
I20260925 16:42:15.362959 18559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:15.373250 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18559
I20260925 16:42:15.373413 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:15.373726 391 external_mini_cluster.cc:1428] Setting key cc520ef8c38c3221f265b9a8e62a03e4
I20260925 16:42:15.378209 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.364846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:44209 (local address 127.0.97.254:45103)
0925 16:42:15.364990 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.364994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.365941 (+ 947us) server_negotiation.cc:407] Connection header received
0925 16:42:15.367479 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.367483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.367541 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.367656 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:15.368931 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.369604 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.370866 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.371124 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.371260 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.374226 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:15.374230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:15.374233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:15.374264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:15.375793 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.376425 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.376428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.376429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.376503 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.376904 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.376909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.376913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.377175 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:15.377311 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.377834 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.377990 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:42:15.378934 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.365223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45103 (local address 127.0.97.193:44209)
0925 16:42:15.365765 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.365811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.367238 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.367834 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.367841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.368218 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:15.368768 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.368784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.369736 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.369739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.370702 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.370712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.371305 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.371310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.371493 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.372095 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:15.372111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:15.374029 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:15.375915 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.375923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.375941 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.376321 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.376595 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.376601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.376606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.376800 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.377327 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:15.377334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:15.377599 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.377757 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.378013 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":123,"threads_started":1}
I20260925 16:42:15.380187 18688 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45103
I20260925 16:42:15.380440 18688 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:15.380936 18688 heartbeater.cc:511] Master 127.0.97.254:45103 requested a full tablet report, sending...
I20260925 16:42:15.382467 18497 ts_manager.cc:194] Registered new tserver with Master: a2b20dbb80214594aef47d49d16a2270 (127.0.97.193:36105)
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:42:15.383646 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:44209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:42:15.402148 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:15.512786 18693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:15.513062 18693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:15.513136 18693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:15.517163 18693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:15.517225 18693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:15.517354 18693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:15.522791 18693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:15.523867 18693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:15.524892 18693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:15.534435 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:15.534456 18701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.534631 18693 server_base.cc:1034] running on GCE node
W20260925 16:42:15.534483 18699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.535219 18693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:15.535918 18693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:15.537081 18693 hybrid_clock.cc:648] HybridClock initialized: now 1790354535537047 us; error 44 us; skew 500 ppm
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:15.539901 18693 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:42:15.540758 18693 webserver.cc:460] Webserver started at http://127.0.97.194:40575/ using document root <none> and password file <none>
I20260925 16:42:15.541378 18693 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:15.541474 18693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:15.541707 18693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:15.543706 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "6f52e073479b4ea68849056fd317d0b9"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "c571f7701eb3e48d55fbd23730519ab5"
server_key_iv: "9e1b7bbb0173f3636d3285d0e27cf040"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.544271 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "6f52e073479b4ea68849056fd317d0b9"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "c571f7701eb3e48d55fbd23730519ab5"
server_key_iv: "9e1b7bbb0173f3636d3285d0e27cf040"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.547988 18693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:42:15.551200 18708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.552973 18693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 16:42:15.553119 18693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "6f52e073479b4ea68849056fd317d0b9"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "c571f7701eb3e48d55fbd23730519ab5"
server_key_iv: "9e1b7bbb0173f3636d3285d0e27cf040"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.553259 18693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:15.572868 18693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:15.576200 18693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:15.576356 18693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:15.576884 18693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:15.577746 18693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:15.577795 18693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.577867 18693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:15.577899 18693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.586494 18693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:41861
I20260925 16:42:15.586517 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:41861 every 8 connection(s)
I20260925 16:42:15.587368 18693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:15.588176 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18693
I20260925 16:42:15.588303 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:15.588585 391 external_mini_cluster.cc:1428] Setting key ef5bdd5a3499cea77fd1f81d1a7bb09f
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:42:15.602516 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.589214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:44833 (local address 127.0.97.254:45103)
0925 16:42:15.589392 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.589396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.590402 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 16:42:15.591880 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.591882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.591932 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.592002 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:15.593480 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.594380 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.595563 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.595962 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.596083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.598779 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:15.598782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:15.598784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:15.598808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:15.600384 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.600881 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.600883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.600885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.600991 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.601378 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.601383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.601390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.601581 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:15.601658 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.602277 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.602376 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 16:42:15.603850 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.589725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45103 (local address 127.0.97.194:44833)
0925 16:42:15.590233 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.590284 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.591645 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.592192 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.592205 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.592757 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:15.593321 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.593339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.594551 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.594554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.595433 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.595442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.596117 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.596120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.596259 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.596955 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:15.596971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:15.598623 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:15.600497 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.600502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.600513 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.600788 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.601100 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.601106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.601109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.601251 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.601687 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:15.601696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:15.602022 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.602228 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.602602 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":166,"threads_started":1}
I20260925 16:42:15.605083 18822 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45103
I20260925 16:42:15.605360 18822 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:15.605966 18822 heartbeater.cc:511] Master 127.0.97.254:45103 requested a full tablet report, sending...
I20260925 16:42:15.607115 18497 ts_manager.cc:194] Registered new tserver with Master: 6f52e073479b4ea68849056fd317d0b9 (127.0.97.194:41861)
I20260925 16:42:15.607709 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:44833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:42:15.620573 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:15.729976 18827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:15.730217 18827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:15.730288 18827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:15.734474 18827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:15.734534 18827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:15.734638 18827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:15.739756 18827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:15.740759 18827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:15.741811 18827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:15.751302 18833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.751385 18827 server_base.cc:1034] running on GCE node
W20260925 16:42:15.751307 18835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:15.751307 18832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:15.752012 18827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:15.752686 18827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:15.753841 18827 hybrid_clock.cc:648] HybridClock initialized: now 1790354535753807 us; error 45 us; skew 500 ppm
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:15.756484 18827 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:42:15.757369 18827 webserver.cc:460] Webserver started at http://127.0.97.195:34781/ using document root <none> and password file <none>
I20260925 16:42:15.757997 18827 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:15.758085 18827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:15.758314 18827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:15.760339 18827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "3ab252590d8b4dd4b6232aadf6a2765a"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "1ac57434a26cc06453c26e6e8c93b63e"
server_key_iv: "57a936f99e8bdd4dd131f0ecab321e88"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.760897 18827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab252590d8b4dd4b6232aadf6a2765a"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "1ac57434a26cc06453c26e6e8c93b63e"
server_key_iv: "57a936f99e8bdd4dd131f0ecab321e88"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.764482 18827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:42:15.767091 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.768392 18827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:15.768529 18827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "3ab252590d8b4dd4b6232aadf6a2765a"
format_stamp: "Formatted at 2026-09-25 16:42:15 on dist-test-slave-kxl8"
server_key: "1ac57434a26cc06453c26e6e8c93b63e"
server_key_iv: "57a936f99e8bdd4dd131f0ecab321e88"
server_key_version: "encryptionkey@0"
I20260925 16:42:15.768646 18827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:15.825842 18827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:15.828778 18827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:15.828998 18827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:15.829631 18827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:15.830536 18827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:15.830610 18827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.830675 18827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:15.830725 18827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:15.839779 18827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:46797
I20260925 16:42:15.839804 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:46797 every 8 connection(s)
I20260925 16:42:15.840636 18827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:15.847134 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 18827
I20260925 16:42:15.847292 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:15.847584 391 external_mini_cluster.cc:1428] Setting key 30ef5e1e8846ea4e79e84444a6b99c14
Sep 25 16:42:15 dist-test-slave-kxl8 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:15.854305 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.842397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:55791 (local address 127.0.97.254:45103)
0925 16:42:15.842522 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.842526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.843340 (+ 814us) server_negotiation.cc:407] Connection header received
0925 16:42:15.844635 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.844638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.844695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.844806 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:15.846014 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.846548 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.847787 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.848124 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.848236 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.850976 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:15.850980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:15.850983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:15.851017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:15.852199 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.852700 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.852705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.852707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.852789 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.853170 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.853173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.853175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.853431 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:15.853540 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.854020 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.854141 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:42:15.855060 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.842774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:45103 (local address 127.0.97.195:55791)
0925 16:42:15.843200 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.843242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.844432 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.844983 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.844989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.845399 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:15.845867 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.845880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.846677 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.846680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.847664 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.847676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.848274 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.848277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.848501 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.849157 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:15.849177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:15.850801 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:15.852315 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.852320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.852333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.852591 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.852889 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:15.852891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:15.852893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:15.853074 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:15.853536 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:15.853542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:15.853781 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.853967 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.854272 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260925 16:42:15.856103 18956 heartbeater.cc:347] Connected to a master server at 127.0.97.254:45103
I20260925 16:42:15.856308 18956 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:15.856726 18956 heartbeater.cc:511] Master 127.0.97.254:45103 requested a full tablet report, sending...
I20260925 16:42:15.857725 18497 ts_manager.cc:194] Registered new tserver with Master: 3ab252590d8b4dd4b6232aadf6a2765a (127.0.97.195:46797)
I20260925 16:42:15.858315 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:55791
I20260925 16:42:15.861155 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:15.870872 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.863047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.97.254:45103)
0925 16:42:15.863172 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.863175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.863354 (+ 179us) server_negotiation.cc:407] Connection header received
0925 16:42:15.863471 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.863474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.863516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.863596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:15.864406 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.864903 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.865814 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.866031 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.866184 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.867819 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:15.867823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:15.867825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:15.867849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:15.869027 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.869586 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.869590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.869592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.869668 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:15.869996 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:15.869999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:15.870001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:15.870198 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:15.870366 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.870625 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.870719 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:15.873678 18497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:15.876602 18497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:15.890767 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.884793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36105 (local address 127.0.0.1:45178)
0925 16:42:15.885480 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.885494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.885567 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.886476 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.886479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.886496 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:15.886579 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.886584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.887962 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.887966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.889091 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.889100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890214 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.890326 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890361 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.890484 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.890602 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":41,"thread_start_us":85,"threads_started":1}
I20260925 16:42:15.890789 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.885767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.0.97.193:36105)
0925 16:42:15.886043 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.886049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.886072 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:42:15.886143 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.886147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.886318 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.886722 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:15.886842 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.887859 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.889214 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890082 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890200 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890358 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.890526 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.890639 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260925 16:42:15.891091 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.886122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.97.194:41861)
0925 16:42:15.886461 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.886474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.886626 (+ 152us) server_negotiation.cc:407] Connection header received
0925 16:42:15.886761 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.886766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.886907 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.887054 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:15.887263 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.888193 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.889611 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890197 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890299 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890435 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.890830 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.890960 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":142,"threads_started":1}
I20260925 16:42:15.891172 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.884772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.97.195:46797)
0925 16:42:15.885199 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:15.885212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:15.885241 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:42:15.885556 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:15.885563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:15.885780 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:15.885956 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:15.886421 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.887614 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.889385 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890105 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890346 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:15.890932 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 16:42:15.891035 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":128,"threads_started":1}
I20260925 16:42:15.890827 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.884636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:46797 (local address 127.0.0.1:60624)
0925 16:42:15.885131 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.885178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.885333 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.886138 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.886144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.886167 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:15.886283 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.886290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.887752 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.887756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.889265 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.889294 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890419 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.890526 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890553 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.890622 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.890686 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":116,"thread_start_us":64,"threads_started":1}
I20260925 16:42:15.891423 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:15.885347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41861 (local address 127.0.0.1:39088)
0925 16:42:15.886518 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:15.886531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:15.886655 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:15.887042 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:15.887045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:15.887062 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:15.887154 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.887160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.888342 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.888345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.889502 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:15.889509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:15.890497 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:15.890501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:15.890586 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:15.890606 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:42:15.890670 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:42:15.890725 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":61,"thread_start_us":86,"threads_started":1}
I20260925 16:42:15.894063 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 (DEFAULT_TABLE table=test-table [id=989df2fca4964bc180bc1dd208f2603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:15.894152 18622 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 (DEFAULT_TABLE table=test-table [id=989df2fca4964bc180bc1dd208f2603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:15.894153 18756 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 (DEFAULT_TABLE table=test-table [id=989df2fca4964bc180bc1dd208f2603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:15.895064 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:15.895133 18622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:15.895133 18756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:15.901410 18973 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Bootstrap starting.
I20260925 16:42:15.902762 18975 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Bootstrap starting.
I20260925 16:42:15.902755 18974 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Bootstrap starting.
I20260925 16:42:15.903820 18973 tablet_bootstrap.cc:654] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:15.904785 18975 tablet_bootstrap.cc:654] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:15.904906 18974 tablet_bootstrap.cc:654] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:15.905074 18973 log.cc:824] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:15.905721 18974 log.cc:824] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:15.905854 18975 log.cc:824] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:15.907289 18973 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: No bootstrap required, opened a new log
I20260925 16:42:15.907511 18973 ts_tablet_manager.cc:1397] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:15.907660 18974 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: No bootstrap required, opened a new log
I20260925 16:42:15.907831 18974 ts_tablet_manager.cc:1397] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:42:15.907853 18975 tablet_bootstrap.cc:492] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: No bootstrap required, opened a new log
I20260925 16:42:15.908013 18975 ts_tablet_manager.cc:1397] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:42:15.910964 18973 raft_consensus.cc:348] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.911227 18973 raft_consensus.cc:374] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:15.911309 18973 raft_consensus.cc:729] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab252590d8b4dd4b6232aadf6a2765a, State: Initialized, Role: FOLLOWER
I20260925 16:42:15.911767 18973 consensus_queue.cc:260] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.912011 18975 raft_consensus.cc:348] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.912290 18975 raft_consensus.cc:374] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:15.912151 18974 raft_consensus.cc:348] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.912357 18974 raft_consensus.cc:374] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:15.912364 18975 raft_consensus.cc:729] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f52e073479b4ea68849056fd317d0b9, State: Initialized, Role: FOLLOWER
I20260925 16:42:15.912418 18974 raft_consensus.cc:729] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b20dbb80214594aef47d49d16a2270, State: Initialized, Role: FOLLOWER
I20260925 16:42:15.912508 18956 heartbeater.cc:503] Master 127.0.97.254:45103 was elected leader, sending a full tablet report...
I20260925 16:42:15.912850 18973 ts_tablet_manager.cc:1428] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:42:15.912792 18975 consensus_queue.cc:260] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.912792 18974 consensus_queue.cc:260] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:15.913491 18688 heartbeater.cc:503] Master 127.0.97.254:45103 was elected leader, sending a full tablet report...
I20260925 16:42:15.913718 18974 ts_tablet_manager.cc:1428] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:42:15.913718 18975 ts_tablet_manager.cc:1428] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:42:15.914026 18822 heartbeater.cc:503] Master 127.0.97.254:45103 was elected leader, sending a full tablet report...
I20260925 16:42:16.084491 18980 raft_consensus.cc:482] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:16.084738 18980 raft_consensus.cc:504] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:16.085901 18980 leader_election.cc:288] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab252590d8b4dd4b6232aadf6a2765a (127.0.97.195:46797), 6f52e073479b4ea68849056fd317d0b9 (127.0.97.194:41861)
W20260925 16:42:16.088429 18823 tablet.cc:2369] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:16.090672 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.086275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:34921 (local address 127.0.97.195:46797)
0925 16:42:16.086493 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.086498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.086831 (+ 333us) server_negotiation.cc:407] Connection header received
0925 16:42:16.087054 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.087057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.087107 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.087176 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:16.087500 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.087991 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.089579 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.090102 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.090174 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.090213 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.090486 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.090555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 16:42:16.090699 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.086189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:46797 (local address 127.0.97.193:34921)
0925 16:42:16.086734 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.086746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:16.086874 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:16.087258 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:16.087261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:16.087278 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:16.087365 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.087374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.088103 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.088108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.089440 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.089454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.090286 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.090289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.090357 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.090372 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 16:42:16.090467 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 16:42:16.090536 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":350,"thread_start_us":125,"threads_started":1}
I20260925 16:42:16.091157 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.086186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41861 (local address 127.0.97.193:44689)
0925 16:42:16.086620 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.086637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:16.086813 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:16.087156 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:16.087159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:16.087174 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:16.087311 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.087318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.088345 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.088348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.089780 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.089788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.090703 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.090706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.090802 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.090824 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:42:16.090942 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:42:16.091016 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":182,"thread_start_us":128,"threads_started":1}
I20260925 16:42:16.091209 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.086264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:44689 (local address 127.0.97.194:41861)
0925 16:42:16.086399 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.086404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.086717 (+ 313us) server_negotiation.cc:407] Connection header received
0925 16:42:16.086975 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.086982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.087041 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.087132 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:16.087448 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.088214 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.089898 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.090490 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.090603 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.090646 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.090996 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.091081 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 16:42:16.091566 18957 tablet.cc:2369] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:16.091812 18776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9cf0afd52740ed93a1c681bdd7b0d9" candidate_uuid: "a2b20dbb80214594aef47d49d16a2270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f52e073479b4ea68849056fd317d0b9" is_pre_election: true
I20260925 16:42:16.091988 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9cf0afd52740ed93a1c681bdd7b0d9" candidate_uuid: "a2b20dbb80214594aef47d49d16a2270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" is_pre_election: true
I20260925 16:42:16.092120 18776 raft_consensus.cc:2454] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b20dbb80214594aef47d49d16a2270 in term 0.
I20260925 16:42:16.092198 18910 raft_consensus.cc:2454] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b20dbb80214594aef47d49d16a2270 in term 0.
I20260925 16:42:16.092643 18577 leader_election.cc:302] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ab252590d8b4dd4b6232aadf6a2765a, 6f52e073479b4ea68849056fd317d0b9, a2b20dbb80214594aef47d49d16a2270; no voters:
I20260925 16:42:16.093081 18980 raft_consensus.cc:2781] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:16.093155 18980 raft_consensus.cc:482] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:16.093194 18980 raft_consensus.cc:3037] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:16.094466 18980 raft_consensus.cc:504] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:16.095191 18980 leader_election.cc:288] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [CANDIDATE]: Term 1 election: Requested vote from peers 3ab252590d8b4dd4b6232aadf6a2765a (127.0.97.195:46797), 6f52e073479b4ea68849056fd317d0b9 (127.0.97.194:41861)
I20260925 16:42:16.095189 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9cf0afd52740ed93a1c681bdd7b0d9" candidate_uuid: "a2b20dbb80214594aef47d49d16a2270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab252590d8b4dd4b6232aadf6a2765a"
I20260925 16:42:16.095257 18776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9cf0afd52740ed93a1c681bdd7b0d9" candidate_uuid: "a2b20dbb80214594aef47d49d16a2270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f52e073479b4ea68849056fd317d0b9"
I20260925 16:42:16.095307 18910 raft_consensus.cc:3037] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:16.095394 18776 raft_consensus.cc:3037] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:16.096649 18910 raft_consensus.cc:2454] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b20dbb80214594aef47d49d16a2270 in term 1.
I20260925 16:42:16.096649 18776 raft_consensus.cc:2454] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b20dbb80214594aef47d49d16a2270 in term 1.
I20260925 16:42:16.096985 18578 leader_election.cc:302] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab252590d8b4dd4b6232aadf6a2765a, a2b20dbb80214594aef47d49d16a2270; no voters:
I20260925 16:42:16.097172 18980 raft_consensus.cc:2781] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:16.097417 18980 raft_consensus.cc:686] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 1 LEADER]: Becoming Leader. State: Replica: a2b20dbb80214594aef47d49d16a2270, State: Running, Role: LEADER
I20260925 16:42:16.097795 18980 consensus_queue.cc:237] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } }
I20260925 16:42:16.100790 18497 catalog_manager.cc:5495] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b20dbb80214594aef47d49d16a2270 (127.0.97.193). New cstate: current_term: 1 leader_uuid: "a2b20dbb80214594aef47d49d16a2270" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b20dbb80214594aef47d49d16a2270" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 36105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 } health_report { overall_health: UNKNOWN } } }
W20260925 16:42:16.114218 18689 tablet.cc:2369] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:16.126120 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.122741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.0.97.193:36105)
0925 16:42:16.122895 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.122899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.122914 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:42:16.123026 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.123028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.123081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.123185 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:16.123383 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.123891 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.124673 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.124902 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.124980 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.125240 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:16.125647 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:16.125758 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.125910 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.125954 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:42:16.132005 18910 raft_consensus.cc:1261] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 1 FOLLOWER]: Refusing update from remote peer a2b20dbb80214594aef47d49d16a2270: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:16.132005 18776 raft_consensus.cc:1261] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 1 FOLLOWER]: Refusing update from remote peer a2b20dbb80214594aef47d49d16a2270: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:16.132705 18984 consensus_queue.cc:1035] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab252590d8b4dd4b6232aadf6a2765a" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:16.132899 18980 consensus_queue.cc:1035] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f52e073479b4ea68849056fd317d0b9" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:16.138689 18987 mvcc.cc:205] Tried to move back new op lower bound from 7333292179988463616 to 7333292179859259392. Current Snapshot: MvccSnapshot[applied={T|T < 7333292179988463616}]
I20260925 16:42:16.140903 18988 mvcc.cc:205] Tried to move back new op lower bound from 7333292179988463616 to 7333292179859259392. Current Snapshot: MvccSnapshot[applied={T|T < 7333292179988463616}]
I20260925 16:42:16.148676 18496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:16.148865 18496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:16.151173 18496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d7298edb3464490873052fbaeadbacd: Sending DeleteTablet for 3 replicas of tablet 8e9cf0afd52740ed93a1c681bdd7b0d9
I20260925 16:42:16.152029 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:60624
I20260925 16:42:16.152040 18756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:39088
I20260925 16:42:16.152127 18622 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:45178
I20260925 16:42:16.152549 18999 tablet_replica.cc:330] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: stopping tablet replica
I20260925 16:42:16.152667 18998 tablet_replica.cc:330] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270: stopping tablet replica
I20260925 16:42:16.152875 18999 raft_consensus.cc:2229] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:16.152920 18998 raft_consensus.cc:2229] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P a2b20dbb80214594aef47d49d16a2270 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:16.153050 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18559
I20260925 16:42:16.153198 18999 raft_consensus.cc:2258] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:16.154805 18999 ts_tablet_manager.cc:1905] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:16.157119 18997 tablet_replica.cc:330] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: stopping tablet replica
I20260925 16:42:16.157208 18999 ts_tablet_manager.cc:1918] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:16.157321 18999 log.cc:1196] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/8e9cf0afd52740ed93a1c681bdd7b0d9
I20260925 16:42:16.157507 18997 raft_consensus.cc:2229] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:16.157707 18999 ts_tablet_manager.cc:1939] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 6f52e073479b4ea68849056fd317d0b9: Deleting consensus metadata
I20260925 16:42:16.157862 18997 raft_consensus.cc:2258] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:16.159116 18483 catalog_manager.cc:4841] TS 6f52e073479b4ea68849056fd317d0b9 (127.0.97.194:41861): tablet 8e9cf0afd52740ed93a1c681bdd7b0d9 (table test-table [id=989df2fca4964bc180bc1dd208f2603e]) successfully deleted
I20260925 16:42:16.159240 18997 ts_tablet_manager.cc:1905] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:16.160358 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18693
W20260925 16:42:16.160351 18483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:36105 (error 108)
I20260925 16:42:16.162211 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.161875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:36105 (local address 127.0.0.1:45200)
0925 16:42:16.162037 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.162112 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 16:42:16.162376 18483 catalog_manager.cc:4584] TS a2b20dbb80214594aef47d49d16a2270 (127.0.97.193:36105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e9cf0afd52740ed93a1c681bdd7b0d9: Network error: Client connection negotiation failed: client connection to 127.0.97.193:36105: connect: Connection refused (error 111)
I20260925 16:42:16.166659 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18827
I20260925 16:42:16.166962 18997 ts_tablet_manager.cc:1918] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:16.167054 18997 log.cc:1196] T 8e9cf0afd52740ed93a1c681bdd7b0d9 P 3ab252590d8b4dd4b6232aadf6a2765a: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/8e9cf0afd52740ed93a1c681bdd7b0d9
I20260925 16:42:16.172014 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 18465
2026-09-25T16:42:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2425 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T16:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:16Z Disabled control of system clock
I20260925 16:42:16.200675 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:40959
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:16.307776 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:16.308040 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:16.308075 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:16.311937 19004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:16.311986 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:16.312002 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:16.312018 19004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:16.312033 19004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:16.316555 19004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:40959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:16.317512 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:16.318773 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:16.327016 19004 server_base.cc:1034] running on GCE node
W20260925 16:42:16.327047 19009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.327155 19012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.327047 19010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:16.327737 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:16.328788 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:16.329982 19004 hybrid_clock.cc:648] HybridClock initialized: now 1790354536329963 us; error 32 us; skew 500 ppm
I20260925 16:42:16.331861 19004 webserver.cc:460] Webserver started at http://127.0.97.254:41957/ using document root <none> and password file <none>
I20260925 16:42:16.332432 19004 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:16.332496 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:16.332723 19004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:16.334772 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d2006f33553d4bbabcdc661d971a28ad"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "920fbf1802137e47f1d7195e5ff20d39"
server_key_iv: "f2442843ce990b3dc8d9191c4863a292"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.335345 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d2006f33553d4bbabcdc661d971a28ad"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "920fbf1802137e47f1d7195e5ff20d39"
server_key_iv: "f2442843ce990b3dc8d9191c4863a292"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.338869 19004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:42:16.341216 19018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.342408 19004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:16.342537 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d2006f33553d4bbabcdc661d971a28ad"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "920fbf1802137e47f1d7195e5ff20d39"
server_key_iv: "f2442843ce990b3dc8d9191c4863a292"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.342638 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:16.359354 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:16.360040 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:16.360250 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:16.367336 19004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:40959
I20260925 16:42:16.367350 19070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:40959 every 8 connection(s)
I20260925 16:42:16.368270 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:16.371273 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:16.376081 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19004
I20260925 16:42:16.376216 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:16.376495 391 external_mini_cluster.cc:1428] Setting key b82595322839546ddbfd337475d82713
I20260925 16:42:16.376791 19071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: Bootstrap starting.
I20260925 16:42:16.379209 19071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:16.380040 19071 log.cc:824] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:16.382138 19071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: No bootstrap required, opened a new log
I20260925 16:42:16.385646 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.377815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.97.254:40959)
0925 16:42:16.378420 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.378434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.378468 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:42:16.379070 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.379085 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.379516 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.379789 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:16.380151 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.381385 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.382513 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.382965 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.383079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.383924 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:16.383929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:16.383940 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:16.384300 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:16.384351 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:16.384369 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:16.384449 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.384612 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.384888 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":140,"threads_started":1}
I20260925 16:42:16.386129 19071 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } }
I20260925 16:42:16.386330 19071 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:16.386385 19071 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2006f33553d4bbabcdc661d971a28ad, State: Initialized, Role: FOLLOWER
I20260925 16:42:16.386849 19071 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } }
I20260925 16:42:16.387001 19071 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:16.387065 19071 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:16.387157 19071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:16.388099 19071 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } }
I20260925 16:42:16.388374 19071 leader_election.cc:302] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2006f33553d4bbabcdc661d971a28ad; no voters:
I20260925 16:42:16.388607 19071 leader_election.cc:288] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:16.388725 19076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:16.388943 19076 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [term 1 LEADER]: Becoming Leader. State: Replica: d2006f33553d4bbabcdc661d971a28ad, State: Running, Role: LEADER
I20260925 16:42:16.389243 19076 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } }
I20260925 16:42:16.389775 19071 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:16.390884 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2006f33553d4bbabcdc661d971a28ad. Latest consensus state: current_term: 1 leader_uuid: "d2006f33553d4bbabcdc661d971a28ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } } }
I20260925 16:42:16.390898 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2006f33553d4bbabcdc661d971a28ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2006f33553d4bbabcdc661d971a28ad" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 40959 } } }
I20260925 16:42:16.391131 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:16.391158 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:16.391934 19086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:16.394835 19086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:16.400024 19086 catalog_manager.cc:1347] Generated new cluster ID: ba0b507d65de4ade990fbe4a3f61402e
I20260925 16:42:16.400098 19086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:16.411968 19086 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:16.412812 19086 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:16.423096 19086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: Generated new TSK 0
I20260925 16:42:16.423779 19086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:16.429137 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--builtin_ntp_servers=127.0.97.212:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:16.542177 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:16.542431 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:16.542469 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:16.546591 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:16.546638 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:16.546729 19095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:16.551517 19095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:16.552542 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:16.553901 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:16.563414 19101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.563395 19103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.563400 19100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:16.563781 19095 server_base.cc:1034] running on GCE node
I20260925 16:42:16.564175 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:16.564780 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:16.565936 19095 hybrid_clock.cc:648] HybridClock initialized: now 1790354536565922 us; error 26 us; skew 500 ppm
I20260925 16:42:16.567656 19095 webserver.cc:460] Webserver started at http://127.0.97.193:44153/ using document root <none> and password file <none>
I20260925 16:42:16.568264 19095 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:16.568324 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:16.568573 19095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:16.570608 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "259c94da3fc94ca3b75a51ab35fc268f"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "c336dba2f5ebacd1a324dfb92bd829dc"
server_key_iv: "82086ef4ea57ed54c032d4d4c54cd8f4"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.571180 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "259c94da3fc94ca3b75a51ab35fc268f"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "c336dba2f5ebacd1a324dfb92bd829dc"
server_key_iv: "82086ef4ea57ed54c032d4d4c54cd8f4"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.574698 19095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:16.576993 19109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.578177 19095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:16.578317 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "259c94da3fc94ca3b75a51ab35fc268f"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "c336dba2f5ebacd1a324dfb92bd829dc"
server_key_iv: "82086ef4ea57ed54c032d4d4c54cd8f4"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.578437 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:16.596478 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:16.597180 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:16.597405 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:16.597993 19095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:16.598996 19095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:16.599041 19095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.599112 19095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:16.599141 19095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.607717 19095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:44279
I20260925 16:42:16.607743 19222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:44279 every 8 connection(s)
I20260925 16:42:16.608675 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:16.608870 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19095
I20260925 16:42:16.608992 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:42:16.609372 391 external_mini_cluster.cc:1428] Setting key e91cf188dfc186fb890ef59301f203f6
I20260925 16:42:16.611680 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--builtin_ntp_servers=127.0.97.212:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:16.619207 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.610650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:53799 (local address 127.0.97.254:40959)
0925 16:42:16.610794 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.610798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.612204 (+ 1406us) server_negotiation.cc:407] Connection header received
0925 16:42:16.613917 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.613920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.613977 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.614063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:16.615041 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.615642 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.616919 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.617185 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.617333 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.618633 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:16.618637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:16.618640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:16.618661 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:16.618680 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:16.618688 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:16.618812 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.618986 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.619052 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:42:16.620108 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.611147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40959 (local address 127.0.97.193:53799)
0925 16:42:16.612040 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.612100 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:16.613689 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:16.614262 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:16.614270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:16.614366 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:16.614880 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.614894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.615800 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.615804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.616774 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.616784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.617394 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.617399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.617584 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.618345 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:16.618347 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:16.618390 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:16.618464 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:16.618814 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:16.618827 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:16.618837 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 16:42:16.618986 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:42:16.619297 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":165,"threads_started":1}
I20260925 16:42:16.621618 19223 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40959
I20260925 16:42:16.621860 19223 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:16.622439 19223 heartbeater.cc:511] Master 127.0.97.254:40959 requested a full tablet report, sending...
I20260925 16:42:16.624042 19035 ts_manager.cc:194] Registered new tserver with Master: 259c94da3fc94ca3b75a51ab35fc268f (127.0.97.193:44279)
I20260925 16:42:16.625613 19035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.193:53799
W20260925 16:42:16.723759 19226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:16.723990 19226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:16.724026 19226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:16.727901 19226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:16.727947 19226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:16.727998 19226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:16.732559 19226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:16.733500 19226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:16.734753 19226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:16.744243 19226 server_base.cc:1034] running on GCE node
W20260925 16:42:16.744167 19231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.744179 19234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.744170 19232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:16.744812 19226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:16.745527 19226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:16.746711 19226 hybrid_clock.cc:648] HybridClock initialized: now 1790354536746672 us; error 59 us; skew 500 ppm
I20260925 16:42:16.748533 19226 webserver.cc:460] Webserver started at http://127.0.97.194:38063/ using document root <none> and password file <none>
I20260925 16:42:16.749087 19226 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:16.749152 19226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:16.749414 19226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:16.751456 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "8ae2f74706384a8ca46a6fd690642ad1"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "4626ae134fdfcd7938ec94ec1578628e"
server_key_iv: "a98073dc36642bbd9f9b14d795207df6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.752027 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "8ae2f74706384a8ca46a6fd690642ad1"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "4626ae134fdfcd7938ec94ec1578628e"
server_key_iv: "a98073dc36642bbd9f9b14d795207df6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.755585 19226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 16:42:16.757948 19240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.759064 19226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:16.759181 19226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "8ae2f74706384a8ca46a6fd690642ad1"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "4626ae134fdfcd7938ec94ec1578628e"
server_key_iv: "a98073dc36642bbd9f9b14d795207df6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.759282 19226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:16.780485 19226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:16.781265 19226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:16.781533 19226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:16.782178 19226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:16.783222 19226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:16.783272 19226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.783339 19226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:16.783373 19226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.792433 19226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:32865
I20260925 16:42:16.792510 19353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:32865 every 8 connection(s)
I20260925 16:42:16.793418 19226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:16.797959 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19226
I20260925 16:42:16.798089 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:16.798373 391 external_mini_cluster.cc:1428] Setting key 6c0c843965f5e75312c6bec63f5248a4
I20260925 16:42:16.800391 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--builtin_ntp_servers=127.0.97.212:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:16.803387 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.795200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:37405 (local address 127.0.97.254:40959)
0925 16:42:16.795350 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.795355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.796368 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 16:42:16.797818 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.797821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.797879 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.797961 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:16.798990 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.799557 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.801336 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.801548 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.801631 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.802815 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:16.802821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:16.802827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:16.802844 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:16.802870 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:16.802883 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:16.802963 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.803106 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.803177 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:42:16.804194 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.795583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40959 (local address 127.0.97.194:37405)
0925 16:42:16.796218 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.796256 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:16.797608 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:16.798129 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:16.798136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:16.798260 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:16.798842 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.798862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.799691 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.799694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.801168 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.801183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.801707 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.801713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.801930 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.802556 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:16.802558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:16.802597 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:16.802658 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:16.802970 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:16.802977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:16.802980 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:42:16.803070 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:42:16.803281 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260925 16:42:16.805418 19354 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40959
I20260925 16:42:16.805677 19354 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:16.806315 19354 heartbeater.cc:511] Master 127.0.97.254:40959 requested a full tablet report, sending...
I20260925 16:42:16.807432 19035 ts_manager.cc:194] Registered new tserver with Master: 8ae2f74706384a8ca46a6fd690642ad1 (127.0.97.194:32865)
I20260925 16:42:16.807983 19035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.194:37405
W20260925 16:42:16.910753 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:16.911043 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:16.911123 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:16.915128 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:16.915202 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:16.915290 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:16.919966 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:16.921015 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:16.922453 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:16.931828 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.931799 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:16.931859 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:16.931921 19357 server_base.cc:1034] running on GCE node
I20260925 16:42:16.932410 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:16.933007 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:16.934186 19357 hybrid_clock.cc:648] HybridClock initialized: now 1790354536934151 us; error 47 us; skew 500 ppm
I20260925 16:42:16.935930 19357 webserver.cc:460] Webserver started at http://127.0.97.195:34007/ using document root <none> and password file <none>
I20260925 16:42:16.936470 19357 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:16.936528 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:16.936682 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:16.938776 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "d57101bda7054a34aa74203fbddb62ae"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "42d20869dc329c4fb961191ea789c53d"
server_key_iv: "1fbd22c42814efdd1e94ab620246cef6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.939285 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "d57101bda7054a34aa74203fbddb62ae"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "42d20869dc329c4fb961191ea789c53d"
server_key_iv: "1fbd22c42814efdd1e94ab620246cef6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.942693 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:42:16.944903 19371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.946221 19357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:16.946327 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "d57101bda7054a34aa74203fbddb62ae"
format_stamp: "Formatted at 2026-09-25 16:42:16 on dist-test-slave-kxl8"
server_key: "42d20869dc329c4fb961191ea789c53d"
server_key_iv: "1fbd22c42814efdd1e94ab620246cef6"
server_key_version: "encryptionkey@0"
I20260925 16:42:16.946455 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:16.964160 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:16.964815 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:16.964963 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:16.965591 19357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:16.966538 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:16.966585 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.966626 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:16.966648 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:16.975067 19357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:37821
I20260925 16:42:16.975142 19484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:37821 every 8 connection(s)
I20260925 16:42:16.975986 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:16.976681 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19357
I20260925 16:42:16.976779 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:16.977052 391 external_mini_cluster.cc:1428] Setting key 68f82243f618b665934b33348da3ef17
I20260925 16:42:16.985160 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.977980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:50589 (local address 127.0.97.254:40959)
0925 16:42:16.978123 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.978128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.979352 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 16:42:16.980718 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.980721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.980765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.980833 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:16.981809 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.982323 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.983518 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.983724 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.983799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.984766 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:16.984768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:16.984770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:16.984782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:16.984800 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:16.984807 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:16.984863 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.985007 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.985053 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:42:16.985982 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.978376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:40959 (local address 127.0.97.195:50589)
0925 16:42:16.979170 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:16.979268 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:16.980534 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:16.980970 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:16.980976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:16.981056 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:16.981667 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.981682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.982444 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.982449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.983417 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:16.983426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.983842 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.983845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:16.983996 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.984532 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:16.984533 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:16.984571 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:16.984631 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:16.984885 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:16.984891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:16.984895 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:42:16.984987 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:42:16.985187 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260925 16:42:16.987021 19485 heartbeater.cc:347] Connected to a master server at 127.0.97.254:40959
I20260925 16:42:16.987259 19485 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:16.987758 19485 heartbeater.cc:511] Master 127.0.97.254:40959 requested a full tablet report, sending...
I20260925 16:42:16.988730 19035 ts_manager.cc:194] Registered new tserver with Master: d57101bda7054a34aa74203fbddb62ae (127.0.97.195:37821)
I20260925 16:42:16.989372 19035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.195:50589
I20260925 16:42:16.990986 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:16.996867 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:16.992980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.0.97.254:40959)
0925 16:42:16.993102 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:16.993107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:16.993287 (+ 180us) server_negotiation.cc:407] Connection header received
0925 16:42:16.993476 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:16.993480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:16.993531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:16.993616 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:16.993909 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.994401 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.995260 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:16.995482 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:16.995592 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:16.996377 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:16.996381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:16.996384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:16.996399 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:16.996423 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:16.996432 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:16.996560 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:16.996697 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 16:42:16.996733 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:16.999696 19035 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:17.002761 19035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:17.016954 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.011813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.97.193:44279)
0925 16:42:17.012258 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.012270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.012295 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:42:17.012368 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.012372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.012534 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.012693 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:17.013007 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014229 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.015548 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.016279 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.016406 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.016605 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.016691 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.016783 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":132,"threads_started":1}
I20260925 16:42:17.016949 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.011069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:44279 (local address 127.0.0.1:57700)
0925 16:42:17.011468 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.011516 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:17.011650 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:17.012680 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:17.012686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:17.012707 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:17.012856 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.012863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014358 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.014361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.015399 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.015413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.016410 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.016413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.016523 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.016560 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 16:42:17.016665 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:42:17.016782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":45,"spinlock_wait_cycles":11008,"thread_start_us":100,"threads_started":1}
I20260925 16:42:17.017241 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.011849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:32865 (local address 127.0.0.1:35110)
0925 16:42:17.012062 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.012075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:17.012157 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:17.013401 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:17.013405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:17.013421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:17.013496 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.013501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014759 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.014762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.015845 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.015852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.016834 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.016838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.016926 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.016957 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:42:17.016998 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 16:42:17.017097 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260925 16:42:17.017592 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.011217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37821 (local address 127.0.0.1:50740)
0925 16:42:17.011514 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.011529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:17.011650 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:17.013158 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:17.013161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:17.013177 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:17.013298 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.013307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014338 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.014341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.015668 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.015677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.017190 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.017196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.017317 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.017339 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:42:17.017412 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:42:17.017476 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2,"spinlock_wait_cycles":3200,"thread_start_us":127,"threads_started":1}
I20260925 16:42:17.017971 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.012015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.97.194:32865)
0925 16:42:17.012978 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.012991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.013013 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:42:17.013083 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.013088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.013252 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.013423 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:17.013586 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014646 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.015964 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.016707 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.017428 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.017626 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.017683 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.017788 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":126,"threads_started":1}
I20260925 16:42:17.018008 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.012245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.97.195:37821)
0925 16:42:17.012545 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.012557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.012579 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:42:17.012651 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.012655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.012861 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.013010 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:17.013436 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.014227 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.015800 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.016589 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.016693 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.016887 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.017741 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.017833 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":133,"threads_started":1}
I20260925 16:42:17.020124 19157 tablet_service.cc:1467] Processing CreateTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 (DEFAULT_TABLE table=test-table [id=bbf9e50db9d9442db5fd923692ee9fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:17.020305 19288 tablet_service.cc:1467] Processing CreateTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 (DEFAULT_TABLE table=test-table [id=bbf9e50db9d9442db5fd923692ee9fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:17.021121 19157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a364588a7bdd4a6a8dd0e350c02c8f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:17.021188 19288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a364588a7bdd4a6a8dd0e350c02c8f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:17.023952 19419 tablet_service.cc:1467] Processing CreateTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 (DEFAULT_TABLE table=test-table [id=bbf9e50db9d9442db5fd923692ee9fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:17.025002 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a364588a7bdd4a6a8dd0e350c02c8f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:17.028239 19502 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Bootstrap starting.
I20260925 16:42:17.029243 19503 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Bootstrap starting.
I20260925 16:42:17.030433 19502 tablet_bootstrap.cc:654] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:17.031303 19502 log.cc:824] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:17.031363 19503 tablet_bootstrap.cc:654] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:17.031338 19504 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Bootstrap starting.
I20260925 16:42:17.032146 19503 log.cc:824] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:17.033581 19502 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: No bootstrap required, opened a new log
I20260925 16:42:17.033761 19502 ts_tablet_manager.cc:1397] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:17.033783 19504 tablet_bootstrap.cc:654] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:17.034155 19503 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: No bootstrap required, opened a new log
I20260925 16:42:17.034353 19503 ts_tablet_manager.cc:1397] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:42:17.034524 19504 log.cc:824] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:17.035989 19504 tablet_bootstrap.cc:492] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: No bootstrap required, opened a new log
I20260925 16:42:17.036177 19504 ts_tablet_manager.cc:1397] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:42:17.038125 19502 raft_consensus.cc:348] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.038396 19502 raft_consensus.cc:374] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:17.038455 19502 raft_consensus.cc:729] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae2f74706384a8ca46a6fd690642ad1, State: Initialized, Role: FOLLOWER
I20260925 16:42:17.038362 19503 raft_consensus.cc:348] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.038594 19503 raft_consensus.cc:374] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:17.038705 19503 raft_consensus.cc:729] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259c94da3fc94ca3b75a51ab35fc268f, State: Initialized, Role: FOLLOWER
I20260925 16:42:17.038916 19502 consensus_queue.cc:260] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.039090 19503 consensus_queue.cc:260] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.039628 19354 heartbeater.cc:503] Master 127.0.97.254:40959 was elected leader, sending a full tablet report...
I20260925 16:42:17.039917 19502 ts_tablet_manager.cc:1428] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:42:17.039907 19503 ts_tablet_manager.cc:1428] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:42:17.039752 19504 raft_consensus.cc:348] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.039986 19504 raft_consensus.cc:374] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:17.040041 19504 raft_consensus.cc:729] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57101bda7054a34aa74203fbddb62ae, State: Initialized, Role: FOLLOWER
I20260925 16:42:17.040220 19223 heartbeater.cc:503] Master 127.0.97.254:40959 was elected leader, sending a full tablet report...
I20260925 16:42:17.040446 19504 consensus_queue.cc:260] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.041394 19504 ts_tablet_manager.cc:1428] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260925 16:42:17.041767 19485 heartbeater.cc:503] Master 127.0.97.254:40959 was elected leader, sending a full tablet report...
W20260925 16:42:17.044284 19355 tablet.cc:2369] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:17.095404 19508 raft_consensus.cc:482] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:17.095640 19508 raft_consensus.cc:504] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.096905 19508 leader_election.cc:288] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 259c94da3fc94ca3b75a51ab35fc268f (127.0.97.193:44279), d57101bda7054a34aa74203fbddb62ae (127.0.97.195:37821)
I20260925 16:42:17.101692 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.097395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:45361 (local address 127.0.97.195:37821)
0925 16:42:17.097527 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.097531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.097747 (+ 216us) server_negotiation.cc:407] Connection header received
0925 16:42:17.097844 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.097846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.097914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.097995 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:17.098232 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.098913 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.100356 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.100955 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.101078 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.101127 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.101502 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.101565 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:42:17.101692 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.097202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:51673 (local address 127.0.97.193:44279)
0925 16:42:17.097369 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.097374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.097392 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:42:17.097470 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.097474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.097532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.097594 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:17.097947 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.098730 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.100356 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.100960 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.101078 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.101127 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.101487 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.101560 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:42:17.101706 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.097080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:44279 (local address 127.0.97.194:51673)
0925 16:42:17.097230 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.097247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:17.097367 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:17.097634 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:17.097637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:17.097663 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:17.097800 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.097808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.098847 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.098859 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.100226 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.100232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.101079 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.101083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.101323 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.101358 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:42:17.101444 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:42:17.101556 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":131}
I20260925 16:42:17.101713 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.097178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:37821 (local address 127.0.97.194:45361)
0925 16:42:17.097634 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.097647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:17.097739 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:17.098013 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:17.098015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:17.098031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:17.098119 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.098123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.099006 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.099009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.100220 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:17.100232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.101092 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.101095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:17.101318 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.101407 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:42:17.101475 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:42:17.101556 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":163,"thread_start_us":134,"threads_started":1}
I20260925 16:42:17.102340 19439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a364588a7bdd4a6a8dd0e350c02c8f78" candidate_uuid: "8ae2f74706384a8ca46a6fd690642ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57101bda7054a34aa74203fbddb62ae" is_pre_election: true
I20260925 16:42:17.102337 19177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a364588a7bdd4a6a8dd0e350c02c8f78" candidate_uuid: "8ae2f74706384a8ca46a6fd690642ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259c94da3fc94ca3b75a51ab35fc268f" is_pre_election: true
I20260925 16:42:17.102658 19177 raft_consensus.cc:2454] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae2f74706384a8ca46a6fd690642ad1 in term 0.
I20260925 16:42:17.102656 19439 raft_consensus.cc:2454] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae2f74706384a8ca46a6fd690642ad1 in term 0.
I20260925 16:42:17.103148 19243 leader_election.cc:302] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae2f74706384a8ca46a6fd690642ad1, d57101bda7054a34aa74203fbddb62ae; no voters:
I20260925 16:42:17.103439 19508 raft_consensus.cc:2781] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:17.103525 19508 raft_consensus.cc:482] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:17.103569 19508 raft_consensus.cc:3037] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:17.104559 19508 raft_consensus.cc:504] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
I20260925 16:42:17.104883 19508 leader_election.cc:288] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 259c94da3fc94ca3b75a51ab35fc268f (127.0.97.193:44279), d57101bda7054a34aa74203fbddb62ae (127.0.97.195:37821)
I20260925 16:42:17.105228 19439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a364588a7bdd4a6a8dd0e350c02c8f78" candidate_uuid: "8ae2f74706384a8ca46a6fd690642ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57101bda7054a34aa74203fbddb62ae"
I20260925 16:42:17.105250 19177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a364588a7bdd4a6a8dd0e350c02c8f78" candidate_uuid: "8ae2f74706384a8ca46a6fd690642ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259c94da3fc94ca3b75a51ab35fc268f"
I20260925 16:42:17.105384 19439 raft_consensus.cc:3037] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:17.105414 19177 raft_consensus.cc:3037] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:17.106654 19177 raft_consensus.cc:2454] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae2f74706384a8ca46a6fd690642ad1 in term 1.
I20260925 16:42:17.106670 19439 raft_consensus.cc:2454] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae2f74706384a8ca46a6fd690642ad1 in term 1.
I20260925 16:42:17.106952 19243 leader_election.cc:302] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae2f74706384a8ca46a6fd690642ad1, d57101bda7054a34aa74203fbddb62ae; no voters:
I20260925 16:42:17.107177 19508 raft_consensus.cc:2781] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:17.107414 19508 raft_consensus.cc:686] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 8ae2f74706384a8ca46a6fd690642ad1, State: Running, Role: LEADER
I20260925 16:42:17.107803 19508 consensus_queue.cc:237] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } }
W20260925 16:42:17.109894 19224 tablet.cc:2369] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:17.111022 19034 catalog_manager.cc:5495] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae2f74706384a8ca46a6fd690642ad1 (127.0.97.194). New cstate: current_term: 1 leader_uuid: "8ae2f74706384a8ca46a6fd690642ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae2f74706384a8ca46a6fd690642ad1" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 32865 } health_report { overall_health: HEALTHY } } }
I20260925 16:42:17.139670 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.136189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.97.194:32865)
0925 16:42:17.136317 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:17.136320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:17.136334 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:42:17.136401 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:17.136403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:17.136466 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:17.136554 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:17.136759 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.137239 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.138221 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:17.138444 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:17.138550 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:17.138843 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:17.139235 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:17.139395 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:17.139487 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 16:42:17.139521 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:17.145442 19439 raft_consensus.cc:1261] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 1 FOLLOWER]: Refusing update from remote peer 8ae2f74706384a8ca46a6fd690642ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:17.145505 19177 raft_consensus.cc:1261] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 1 FOLLOWER]: Refusing update from remote peer 8ae2f74706384a8ca46a6fd690642ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:17.146221 19512 consensus_queue.cc:1035] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259c94da3fc94ca3b75a51ab35fc268f" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:17.146414 19508 consensus_queue.cc:1035] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57101bda7054a34aa74203fbddb62ae" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:17.156780 19516 mvcc.cc:205] Tried to move back new op lower bound from 7333292184140443648 to 7333292183996125184. Current Snapshot: MvccSnapshot[applied={T|T < 7333292184140443648}]
I20260925 16:42:17.164855 19034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:17.165041 19034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:17.167493 19034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2006f33553d4bbabcdc661d971a28ad: Sending DeleteTablet for 3 replicas of tablet a364588a7bdd4a6a8dd0e350c02c8f78
I20260925 16:42:17.168222 19157 tablet_service.cc:1514] Processing DeleteTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:17 UTC) from {username='slave'} at 127.0.0.1:57700
I20260925 16:42:17.168434 19419 tablet_service.cc:1514] Processing DeleteTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:17 UTC) from {username='slave'} at 127.0.0.1:50740
I20260925 16:42:17.168541 19288 tablet_service.cc:1514] Processing DeleteTablet for tablet a364588a7bdd4a6a8dd0e350c02c8f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:17 UTC) from {username='slave'} at 127.0.0.1:35110
I20260925 16:42:17.168795 19524 tablet_replica.cc:330] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f: stopping tablet replica
I20260925 16:42:17.168964 19525 tablet_replica.cc:330] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: stopping tablet replica
I20260925 16:42:17.169133 19524 raft_consensus.cc:2229] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:17.169404 19525 raft_consensus.cc:2229] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:17.169451 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19095
I20260925 16:42:17.169510 19524 raft_consensus.cc:2258] T a364588a7bdd4a6a8dd0e350c02c8f78 P 259c94da3fc94ca3b75a51ab35fc268f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:17.171087 19525 raft_consensus.cc:2258] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:17.172732 19525 ts_tablet_manager.cc:1905] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:17.173481 19526 tablet_replica.cc:330] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: stopping tablet replica
I20260925 16:42:17.173799 19526 raft_consensus.cc:2229] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:17.174229 19526 raft_consensus.cc:2258] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:17.175719 19526 ts_tablet_manager.cc:1905] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:17.176265 19525 ts_tablet_manager.cc:1918] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:17.176370 19525 log.cc:1196] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/a364588a7bdd4a6a8dd0e350c02c8f78
I20260925 16:42:17.176702 19525 ts_tablet_manager.cc:1939] T a364588a7bdd4a6a8dd0e350c02c8f78 P d57101bda7054a34aa74203fbddb62ae: Deleting consensus metadata
I20260925 16:42:17.178022 19021 catalog_manager.cc:4841] TS d57101bda7054a34aa74203fbddb62ae (127.0.97.195:37821): tablet a364588a7bdd4a6a8dd0e350c02c8f78 (table test-table [id=bbf9e50db9d9442db5fd923692ee9fa3]) successfully deleted
W20260925 16:42:17.178210 19019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:44279 (error 108)
I20260925 16:42:17.178741 19526 ts_tablet_manager.cc:1918] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:17.178910 19526 log.cc:1196] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/a364588a7bdd4a6a8dd0e350c02c8f78
I20260925 16:42:17.179221 19526 ts_tablet_manager.cc:1939] T a364588a7bdd4a6a8dd0e350c02c8f78 P 8ae2f74706384a8ca46a6fd690642ad1: Deleting consensus metadata
I20260925 16:42:17.179311 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.178972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:44279 (local address 127.0.0.1:57708)
0925 16:42:17.179151 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.179208 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 16:42:17.179484 19019 catalog_manager.cc:4584] TS 259c94da3fc94ca3b75a51ab35fc268f (127.0.97.193:44279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a364588a7bdd4a6a8dd0e350c02c8f78: Network error: Client connection negotiation failed: client connection to 127.0.97.193:44279: connect: Connection refused (error 111)
I20260925 16:42:17.179950 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19226
I20260925 16:42:17.185005 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19357
I20260925 16:42:17.185804 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:17.185178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:32865 (local address 127.0.0.1:35134)
0925 16:42:17.185637 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:17.185701 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:32865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260925 16:42:17.190730 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19004
2026-09-25T16:42:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:17 dist-test-slave-kxl8 krb5kdc[19530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:17 dist-test-slave-kxl8 krb5kdc[19530](info): set up 2 sockets
Sep 25 16:42:17 dist-test-slave-kxl8 krb5kdc[19530](info): commencing operation
krb5kdc: starting...
W20260925 16:42:18.221221 391 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:18.343887 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38913
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:18.451809 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:18.452065 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:18.452105 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:18.455969 19543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:18.456028 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:18.456048 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:18.456063 19543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:18.456079 19543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:18.461212 19543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:18.462303 19543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:18.463271 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:18.471909 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:18.471954 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:18.471939 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:18.472127 19543 server_base.cc:1034] running on GCE node
I20260925 16:42:18.472641 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:18.473809 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:18.474972 19543 hybrid_clock.cc:648] HybridClock initialized: now 1790354538474953 us; error 31 us; skew 500 ppm
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:18.477882 19543 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:18.478775 19543 webserver.cc:460] Webserver started at http://127.0.97.254:43969/ using document root <none> and password file <none>
I20260925 16:42:18.479400 19543 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:18.479465 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:18.479710 19543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:18.481806 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "dbbf95507a114c7fb6b8a46c8c028141"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "be76a4e7315802fb7e2e70e1127bd4a1"
server_key_iv: "649b29e8a38e5d7c73519c374abbbe84"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.482388 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "dbbf95507a114c7fb6b8a46c8c028141"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "be76a4e7315802fb7e2e70e1127bd4a1"
server_key_iv: "649b29e8a38e5d7c73519c374abbbe84"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.486040 19543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:18.488343 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:18.489639 19543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 16:42:18.489773 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "dbbf95507a114c7fb6b8a46c8c028141"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "be76a4e7315802fb7e2e70e1127bd4a1"
server_key_iv: "649b29e8a38e5d7c73519c374abbbe84"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.489887 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:18.517627 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:18.520587 19543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:18.520793 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:18.527940 19543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:38913
I20260925 16:42:18.527967 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:38913 every 8 connection(s)
I20260925 16:42:18.528882 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:18.529778 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19543
I20260925 16:42:18.529909 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:18.530190 391 external_mini_cluster.cc:1428] Setting key 945c8ecd1b7228d154045acb3851fe8b
I20260925 16:42:18.531924 19611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:18.537331 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: Bootstrap starting.
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:18.539647 19611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:18.540495 19611 log.cc:824] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:18.542686 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: No bootstrap required, opened a new log
I20260925 16:42:18.543922 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:18.531618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.97.254:38913)
0925 16:42:18.532086 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:18.532102 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:18.532138 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:18.532765 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:18.532781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:18.533110 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:18.533480 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:18.534262 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.535254 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.536129 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.536600 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.536763 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:18.539310 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:18.539315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:18.539327 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:18.539361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:18.541118 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:18.541636 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:18.541640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:18.541645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:18.541756 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:18.542041 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:18.542043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:18.542044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:18.542412 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:18.542562 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:18.542904 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:42:18.543214 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260925 16:42:18.546216 19611 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } }
I20260925 16:42:18.546429 19611 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:18.546463 19611 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbf95507a114c7fb6b8a46c8c028141, State: Initialized, Role: FOLLOWER
I20260925 16:42:18.546867 19611 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } }
I20260925 16:42:18.546993 19611 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:18.547070 19611 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:18.547185 19611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:18.548089 19611 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } }
I20260925 16:42:18.548434 19611 leader_election.cc:302] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbbf95507a114c7fb6b8a46c8c028141; no voters:
I20260925 16:42:18.548784 19611 leader_election.cc:288] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:18.548906 19616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:18.549134 19616 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [term 1 LEADER]: Becoming Leader. State: Replica: dbbf95507a114c7fb6b8a46c8c028141, State: Running, Role: LEADER
I20260925 16:42:18.549505 19616 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } }
I20260925 16:42:18.550053 19611 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:18.551167 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbbf95507a114c7fb6b8a46c8c028141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } } }
I20260925 16:42:18.551313 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:18.551730 19625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:18.551717 19617 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbbf95507a114c7fb6b8a46c8c028141. Latest consensus state: current_term: 1 leader_uuid: "dbbf95507a114c7fb6b8a46c8c028141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbf95507a114c7fb6b8a46c8c028141" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 38913 } } }
I20260925 16:42:18.551991 19617 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:18.554868 19625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:18.560148 19625 catalog_manager.cc:1347] Generated new cluster ID: 64dea4c325404357891014879b7b97fd
I20260925 16:42:18.560213 19625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:18.575479 19625 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:18.576736 19625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:18.582450 19625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: Generated new TSK 0
I20260925 16:42:18.583268 19625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
Entry for principal kudu/127.0.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab.
I20260925 16:42:18.612987 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:18.724520 19637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:18.724776 19637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:18.724818 19637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:18.729025 19637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:18.729081 19637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:18.729175 19637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:18.734344 19637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.193.keytab
--principal=kudu/127.0.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:18.735332 19637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:18.736317 19637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:18.745862 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:18.745842 19643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:18.745842 19645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:18.745935 19637 server_base.cc:1034] running on GCE node
I20260925 16:42:18.746493 19637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:18.747084 19637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:18.748278 19637 hybrid_clock.cc:648] HybridClock initialized: now 1790354538748254 us; error 32 us; skew 500 ppm
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:18.751082 19637 init.cc:377] Logged in from keytab as kudu/127.0.97.193@KRBTEST.COM (short username kudu)
I20260925 16:42:18.752044 19637 webserver.cc:460] Webserver started at http://127.0.97.193:46811/ using document root <none> and password file <none>
I20260925 16:42:18.752549 19637 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:18.752605 19637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:18.752760 19637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:18.754777 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "38d470acc24540dcbb3e77f8b57d3d42"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "fdac668317b4537322f6c0f239d132b4"
server_key_iv: "0c917aed1a666ffb298666559512f7e9"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.755263 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "38d470acc24540dcbb3e77f8b57d3d42"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "fdac668317b4537322f6c0f239d132b4"
server_key_iv: "0c917aed1a666ffb298666559512f7e9"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.758868 19637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:42:18.761112 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:18.762385 19637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:18.762530 19637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "38d470acc24540dcbb3e77f8b57d3d42"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "fdac668317b4537322f6c0f239d132b4"
server_key_iv: "0c917aed1a666ffb298666559512f7e9"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.762637 19637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:18.785620 19637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:18.788511 19637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:18.788709 19637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:18.789338 19637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:18.790165 19637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:18.790211 19637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:18.790279 19637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:18.790309 19637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:18.799571 19637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:45847
I20260925 16:42:18.799602 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:45847 every 8 connection(s)
I20260925 16:42:18.800457 19637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:18.809392 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19637
I20260925 16:42:18.809528 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.193@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:18.809867 391 external_mini_cluster.cc:1428] Setting key d7864ca93d9e795908dcead813fb189e
I20260925 16:42:18.814177 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:18.802229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:43233 (local address 127.0.97.254:38913)
0925 16:42:18.802375 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:18.802378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:18.803236 (+ 858us) server_negotiation.cc:407] Connection header received
0925 16:42:18.804453 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:18.804456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:18.804503 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:18.804579 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:18.805808 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.806355 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.807428 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.807670 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.807748 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:18.810332 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:18.810336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:18.810338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:18.810368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:18.811909 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:18.812414 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:18.812419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:18.812420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:18.812536 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:18.812915 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:18.812920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:18.812923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:18.813166 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:18.813313 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:18.813790 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 16:42:18.813931 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:42:18.814833 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:18.802650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38913 (local address 127.0.97.193:43233)
0925 16:42:18.803074 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:18.803116 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:18.804247 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:18.804743 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:18.804750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:18.805108 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:18.805673 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:18.805686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.806476 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.806478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:18.807326 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:18.807334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:18.807795 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:18.807797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:18.807937 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:18.808533 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:18.808549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:18.810167 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:18.812026 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:18.812032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:18.812045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:18.812304 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:18.812640 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:18.812646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:18.812649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:18.812818 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:18.813271 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:18.813315 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:18.813551 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 16:42:18.813708 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 16:42:18.813952 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260925 16:42:18.816211 19766 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38913
I20260925 16:42:18.816490 19766 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:18.817117 19766 heartbeater.cc:511] Master 127.0.97.254:38913 requested a full tablet report, sending...
I20260925 16:42:18.818697 19575 ts_manager.cc:194] Registered new tserver with Master: 38d470acc24540dcbb3e77f8b57d3d42 (127.0.97.193:45847)
WARNING: no policy specified for kudu/127.0.97.194@KRBTEST.COM; defaulting to no policy
I20260925 16:42:18.819861 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.193@KRBTEST.COM'} at 127.0.97.193:43233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
Entry for principal kudu/127.0.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab.
I20260925 16:42:18.836580 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:18.945604 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:18.945852 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:18.945896 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:18.950009 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:18.950069 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:18.950163 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:18.955636 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.194.keytab
--principal=kudu/127.0.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:18.956710 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:18.957808 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:18.968519 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:18.968555 19771 server_base.cc:1034] running on GCE node
W20260925 16:42:18.968569 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:18.968526 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:18.969221 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:18.969921 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:18.971081 19771 hybrid_clock.cc:648] HybridClock initialized: now 1790354538971065 us; error 27 us; skew 500 ppm
Sep 25 16:42:18 dist-test-slave-kxl8 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:18.973941 19771 init.cc:377] Logged in from keytab as kudu/127.0.97.194@KRBTEST.COM (short username kudu)
I20260925 16:42:18.974771 19771 webserver.cc:460] Webserver started at http://127.0.97.194:44809/ using document root <none> and password file <none>
I20260925 16:42:18.975282 19771 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:18.975356 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:18.975521 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:18.977578 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "22d25e01782546ada3eb8e796842b114"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "388e40de935f304f5354b9d9547d88f2"
server_key_iv: "1b30c52995419215fb325da6f815afec"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.978071 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "22d25e01782546ada3eb8e796842b114"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "388e40de935f304f5354b9d9547d88f2"
server_key_iv: "1b30c52995419215fb325da6f815afec"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.981664 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:18.984054 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:18.985378 19771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:18.985481 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "22d25e01782546ada3eb8e796842b114"
format_stamp: "Formatted at 2026-09-25 16:42:18 on dist-test-slave-kxl8"
server_key: "388e40de935f304f5354b9d9547d88f2"
server_key_iv: "1b30c52995419215fb325da6f815afec"
server_key_version: "encryptionkey@0"
I20260925 16:42:18.985567 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:19.004004 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:19.007076 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:19.007236 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:19.007786 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:19.008572 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:19.008618 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.008659 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:19.008674 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.017731 19771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:41435
I20260925 16:42:19.017763 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:41435 every 8 connection(s)
I20260925 16:42:19.018638 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:19.022650 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19771
I20260925 16:42:19.022818 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:19.023176 391 external_mini_cluster.cc:1428] Setting key 12a46af4b9751a65797e93f37e57a2d8
Sep 25 16:42:19 dist-test-slave-kxl8 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.194@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.195@KRBTEST.COM; defaulting to no policy
I20260925 16:42:19.033421 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.020430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:58247 (local address 127.0.97.254:38913)
0925 16:42:19.020576 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.020579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.021449 (+ 870us) server_negotiation.cc:407] Connection header received
0925 16:42:19.022805 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.022811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.022870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.022965 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:19.024422 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.025018 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.026704 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.027030 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.027146 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.029722 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:19.029726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:19.029729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:19.029755 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:19.031277 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.031792 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.031796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.031798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.031885 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.032280 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.032283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.032284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.032477 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:19.032586 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.033094 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.033217 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:42:19.034219 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.020801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38913 (local address 127.0.97.194:58247)
0925 16:42:19.021237 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.021326 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.022535 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.023192 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.023203 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.023680 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:19.024249 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.024262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.025147 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.025153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.026576 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.026585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.027184 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.027188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.027358 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.027978 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:19.027994 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:19.029556 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:19.031397 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:19.031403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:19.031425 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:19.031681 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:19.031984 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:19.031987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:19.031989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:19.032171 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:19.032584 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:19.032591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:19.032849 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.033036 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.033384 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.195@KRBTEST.COM" created.
I20260925 16:42:19.035483 19900 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38913
I20260925 16:42:19.035751 19900 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:19.036378 19900 heartbeater.cc:511] Master 127.0.97.254:38913 requested a full tablet report, sending...
I20260925 16:42:19.037647 19575 ts_manager.cc:194] Registered new tserver with Master: 22d25e01782546ada3eb8e796842b114 (127.0.97.194:41435)
I20260925 16:42:19.038237 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.194@KRBTEST.COM'} at 127.0.97.194:58247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
Entry for principal kudu/127.0.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab.
I20260925 16:42:19.052425 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:19.162534 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:19.162833 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:19.162926 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:19.167804 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:19.167991 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:19.168080 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:19.173525 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/krb5kdc/kudu_127.0.97.195.keytab
--principal=kudu/127.0.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:38913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:19.174543 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:19.175530 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:19.184859 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:19.184834 19910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:19.184880 19911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:19.185163 19905 server_base.cc:1034] running on GCE node
I20260925 16:42:19.185730 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:19.186398 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:19.187556 19905 hybrid_clock.cc:648] HybridClock initialized: now 1790354539187540 us; error 26 us; skew 500 ppm
Sep 25 16:42:19 dist-test-slave-kxl8 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:19.190500 19905 init.cc:377] Logged in from keytab as kudu/127.0.97.195@KRBTEST.COM (short username kudu)
I20260925 16:42:19.191329 19905 webserver.cc:460] Webserver started at http://127.0.97.195:38169/ using document root <none> and password file <none>
I20260925 16:42:19.191912 19905 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:19.191973 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:19.192210 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:19.194393 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "3f355744092a43efbefbed3e1149b711"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "3b98cf116abfe101f5bd3526b5caf533"
server_key_iv: "8d777c057819fab6a10777db8698ae16"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.194939 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "3f355744092a43efbefbed3e1149b711"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "3b98cf116abfe101f5bd3526b5caf533"
server_key_iv: "8d777c057819fab6a10777db8698ae16"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.198629 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:19.200961 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.202306 19905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:19.202441 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "3f355744092a43efbefbed3e1149b711"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "3b98cf116abfe101f5bd3526b5caf533"
server_key_iv: "8d777c057819fab6a10777db8698ae16"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.202546 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:19.217237 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:19.220572 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:19.220775 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:19.221400 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:19.222190 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:19.222235 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.222307 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:19.222339 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.231163 19905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:41707
I20260925 16:42:19.231199 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:41707 every 8 connection(s)
I20260925 16:42:19.232005 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:19.239145 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 19905
I20260925 16:42:19.239286 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:19.239620 391 external_mini_cluster.cc:1428] Setting key 11b2e53b4095cb2bdf971f0c9fe0df19
Sep 25 16:42:19 dist-test-slave-kxl8 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.195@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:19.245680 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.233653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:43551 (local address 127.0.97.254:38913)
0925 16:42:19.233788 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.233792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.234708 (+ 916us) server_negotiation.cc:407] Connection header received
0925 16:42:19.235928 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.235931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.235978 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.236067 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:19.237157 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.237781 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.238836 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.239065 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.239140 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.242237 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:19.242240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:19.242242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:19.242264 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:19.243756 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.244193 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.244195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.244197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.244316 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.244647 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.244650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.244652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.244840 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:19.244926 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.245449 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.245560 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:42:19.246496 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.234121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:38913 (local address 127.0.97.195:43551)
0925 16:42:19.234538 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.234578 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.235718 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.236225 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.236232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.236560 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:19.237020 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.237033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.237889 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.237892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.238733 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.238741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.239216 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.239222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.239430 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.240074 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:19.240089 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:19.242097 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:19.243852 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:19.243856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:19.243868 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:19.244105 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:19.244395 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:19.244397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:19.244398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:19.244537 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:19.244936 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:19.244941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:19.245161 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.245384 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.245645 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260925 16:42:19.247598 20034 heartbeater.cc:347] Connected to a master server at 127.0.97.254:38913
I20260925 16:42:19.247833 20034 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:19.248307 20034 heartbeater.cc:511] Master 127.0.97.254:38913 requested a full tablet report, sending...
I20260925 16:42:19.249402 19575 ts_manager.cc:194] Registered new tserver with Master: 3f355744092a43efbefbed3e1149b711 (127.0.97.195:41707)
I20260925 16:42:19.249938 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.195@KRBTEST.COM'} at 127.0.97.195:43551
I20260925 16:42:19.253523 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:19.262454 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.255349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.97.254:38913)
0925 16:42:19.255516 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.255522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.255656 (+ 134us) server_negotiation.cc:407] Connection header received
0925 16:42:19.255738 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.255740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.255777 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.255841 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:19.256453 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.256936 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.257921 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.258129 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.258208 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.259429 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:19.259432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:19.259434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:19.259454 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:19.260888 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.261243 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.261245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.261246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.261401 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:19.261633 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:19.261636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:19.261637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:19.261832 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:19.261934 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.262126 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.262215 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:42:19.265206 19575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:19.268268 19575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:19.282814 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.276607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:41707 (local address 127.0.0.1:42670)
0925 16:42:19.277442 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.277495 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.277620 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.278316 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.278322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.278346 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:19.278497 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.278503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.279898 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.279901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.281378 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.281388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282254 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.282371 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282409 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.282520 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.282643 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260925 16:42:19.282979 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.277096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41435 (local address 127.0.0.1:60344)
0925 16:42:19.278102 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.278115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.278219 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.278879 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.278882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.278900 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:19.278995 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.279001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.280324 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.280328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.281547 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.281563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282542 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.282638 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282657 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.282731 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.282798 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":27,"thread_start_us":133,"threads_started":1}
I20260925 16:42:19.283118 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.276758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:45847 (local address 127.0.0.1:50476)
0925 16:42:19.277841 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.277855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.277924 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.278647 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.278653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.278674 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:19.278776 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.278782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.279825 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.279830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.281316 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.281330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282677 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.282802 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282825 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.282898 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.282993 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":344,"thread_start_us":73,"threads_started":1}
I20260925 16:42:19.283437 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.277341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.97.195:41707)
0925 16:42:19.277803 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.277815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.277837 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:42:19.277934 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.277942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.278128 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.278293 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:19.278610 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.279750 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.281523 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282129 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282254 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282415 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.283161 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.283282 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":161,"threads_started":1}
I20260925 16:42:19.283519 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.277535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.97.194:41435)
0925 16:42:19.278017 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.278029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.278224 (+ 195us) server_negotiation.cc:407] Connection header received
0925 16:42:19.278386 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.278391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.278533 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.278650 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:19.279118 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.280192 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.281666 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282413 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282519 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282626 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.282776 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.282874 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":122,"threads_started":1}
I20260925 16:42:19.283677 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.276947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.97.193:45847)
0925 16:42:19.278198 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.278212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.278234 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:42:19.278311 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.278318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.278524 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.278667 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:19.278897 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.279697 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.281502 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.282348 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.282469 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.282612 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.283420 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.283520 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":76,"threads_started":1}
I20260925 16:42:19.286152 19834 tablet_service.cc:1467] Processing CreateTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f (DEFAULT_TABLE table=test-table [id=dcbbfa7e00574f919fa7360124e29a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:19.286281 19968 tablet_service.cc:1467] Processing CreateTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f (DEFAULT_TABLE table=test-table [id=dcbbfa7e00574f919fa7360124e29a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:19.286516 19700 tablet_service.cc:1467] Processing CreateTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f (DEFAULT_TABLE table=test-table [id=dcbbfa7e00574f919fa7360124e29a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:19.287180 19834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:19.287272 19968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:19.287300 19700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:19.293715 20051 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Bootstrap starting.
I20260925 16:42:19.294777 20053 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Bootstrap starting.
I20260925 16:42:19.294930 20052 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Bootstrap starting.
I20260925 16:42:19.296852 20051 tablet_bootstrap.cc:654] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:19.296988 20053 tablet_bootstrap.cc:654] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:19.296988 20052 tablet_bootstrap.cc:654] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:19.297710 20051 log.cc:824] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:19.298293 20053 log.cc:824] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:19.298293 20052 log.cc:824] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:19.299773 20051 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: No bootstrap required, opened a new log
I20260925 16:42:19.300000 20051 ts_tablet_manager.cc:1397] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:42:19.300122 20052 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: No bootstrap required, opened a new log
I20260925 16:42:19.300375 20052 ts_tablet_manager.cc:1397] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:19.300503 20053 tablet_bootstrap.cc:492] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: No bootstrap required, opened a new log
I20260925 16:42:19.300710 20053 ts_tablet_manager.cc:1397] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:19.303186 20051 raft_consensus.cc:348] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.303447 20051 raft_consensus.cc:374] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:19.303521 20051 raft_consensus.cc:729] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d25e01782546ada3eb8e796842b114, State: Initialized, Role: FOLLOWER
I20260925 16:42:19.304005 20051 consensus_queue.cc:260] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.304334 20052 raft_consensus.cc:348] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.304363 20053 raft_consensus.cc:348] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.304610 20053 raft_consensus.cc:374] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:19.304656 20053 raft_consensus.cc:729] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f355744092a43efbefbed3e1149b711, State: Initialized, Role: FOLLOWER
I20260925 16:42:19.304744 19900 heartbeater.cc:503] Master 127.0.97.254:38913 was elected leader, sending a full tablet report...
I20260925 16:42:19.305017 20051 ts_tablet_manager.cc:1428] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260925 16:42:19.305058 20053 consensus_queue.cc:260] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.305423 20052 raft_consensus.cc:374] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:19.305534 20052 raft_consensus.cc:729] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d470acc24540dcbb3e77f8b57d3d42, State: Initialized, Role: FOLLOWER
I20260925 16:42:19.305859 20034 heartbeater.cc:503] Master 127.0.97.254:38913 was elected leader, sending a full tablet report...
I20260925 16:42:19.306062 20053 ts_tablet_manager.cc:1428] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260925 16:42:19.306052 20052 consensus_queue.cc:260] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.306682 19766 heartbeater.cc:503] Master 127.0.97.254:38913 was elected leader, sending a full tablet report...
I20260925 16:42:19.307020 20052 ts_tablet_manager.cc:1428] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 16:42:19.482897 20035 tablet.cc:2369] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:19.497092 20058 raft_consensus.cc:482] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:19.497346 20058 raft_consensus.cc:504] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.498584 20058 leader_election.cc:288] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38d470acc24540dcbb3e77f8b57d3d42 (127.0.97.193:45847), 22d25e01782546ada3eb8e796842b114 (127.0.97.194:41435)
I20260925 16:42:19.502468 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.498866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41435 (local address 127.0.97.195:52659)
0925 16:42:19.499225 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.499239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.499339 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.499610 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.499612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.499626 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:19.499762 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.499768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.500479 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.500481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.501382 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.501394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.502011 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.502014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.502145 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.502163 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.502223 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.502306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":21,"thread_start_us":138,"threads_started":1}
I20260925 16:42:19.502519 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.498897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:59935 (local address 127.0.97.193:45847)
0925 16:42:19.499010 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.499016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.499179 (+ 163us) server_negotiation.cc:407] Connection header received
0925 16:42:19.499260 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.499266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.499325 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.499415 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:19.499704 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.500265 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.501484 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.501887 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.502014 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.502073 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.502319 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.502396 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:19.502583 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.498993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:52659 (local address 127.0.97.194:41435)
0925 16:42:19.499135 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.499139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.499324 (+ 185us) server_negotiation.cc:407] Connection header received
0925 16:42:19.499460 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.499463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.499510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.499591 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:19.499870 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.500365 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.501524 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.501891 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.502016 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.502073 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.502260 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.502459 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:42:19.502596 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.498799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:45847 (local address 127.0.97.195:59935)
0925 16:42:19.499001 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.499017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:19.499123 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:19.499441 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:19.499443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:19.499462 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:19.499592 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.499600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.500368 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.500372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.501356 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:19.501374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.502004 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.502008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:19.502175 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.502192 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:42:19.502288 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:42:19.502423 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":141}
I20260925 16:42:19.503626 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c98c76f9cf74ed2b05a60624dcd3c5f" candidate_uuid: "3f355744092a43efbefbed3e1149b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d470acc24540dcbb3e77f8b57d3d42" is_pre_election: true
I20260925 16:42:19.503626 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c98c76f9cf74ed2b05a60624dcd3c5f" candidate_uuid: "3f355744092a43efbefbed3e1149b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22d25e01782546ada3eb8e796842b114" is_pre_election: true
I20260925 16:42:19.503942 19854 raft_consensus.cc:2454] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f355744092a43efbefbed3e1149b711 in term 0.
I20260925 16:42:19.503937 19720 raft_consensus.cc:2454] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f355744092a43efbefbed3e1149b711 in term 0.
I20260925 16:42:19.504470 19922 leader_election.cc:302] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22d25e01782546ada3eb8e796842b114, 38d470acc24540dcbb3e77f8b57d3d42, 3f355744092a43efbefbed3e1149b711; no voters:
I20260925 16:42:19.504783 20058 raft_consensus.cc:2781] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:19.504876 20058 raft_consensus.cc:482] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:19.504945 20058 raft_consensus.cc:3037] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:19.506176 20058 raft_consensus.cc:504] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.506553 20058 leader_election.cc:288] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [CANDIDATE]: Term 1 election: Requested vote from peers 38d470acc24540dcbb3e77f8b57d3d42 (127.0.97.193:45847), 22d25e01782546ada3eb8e796842b114 (127.0.97.194:41435)
I20260925 16:42:19.506891 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c98c76f9cf74ed2b05a60624dcd3c5f" candidate_uuid: "3f355744092a43efbefbed3e1149b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22d25e01782546ada3eb8e796842b114"
I20260925 16:42:19.506943 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c98c76f9cf74ed2b05a60624dcd3c5f" candidate_uuid: "3f355744092a43efbefbed3e1149b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d470acc24540dcbb3e77f8b57d3d42"
I20260925 16:42:19.507007 19854 raft_consensus.cc:3037] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:19.507097 19720 raft_consensus.cc:3037] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:19.508512 19720 raft_consensus.cc:2454] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f355744092a43efbefbed3e1149b711 in term 1.
I20260925 16:42:19.508515 19854 raft_consensus.cc:2454] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f355744092a43efbefbed3e1149b711 in term 1.
I20260925 16:42:19.508831 19921 leader_election.cc:302] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22d25e01782546ada3eb8e796842b114, 38d470acc24540dcbb3e77f8b57d3d42, 3f355744092a43efbefbed3e1149b711; no voters:
I20260925 16:42:19.509006 20058 raft_consensus.cc:2781] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:19.509209 20058 raft_consensus.cc:686] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 1 LEADER]: Becoming Leader. State: Replica: 3f355744092a43efbefbed3e1149b711, State: Running, Role: LEADER
I20260925 16:42:19.509588 20058 consensus_queue.cc:237] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } }
I20260925 16:42:19.512785 19573 catalog_manager.cc:5495] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f355744092a43efbefbed3e1149b711 (127.0.97.195). New cstate: current_term: 1 leader_uuid: "3f355744092a43efbefbed3e1149b711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f355744092a43efbefbed3e1149b711" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 } health_report { overall_health: UNKNOWN } } }
W20260925 16:42:19.519783 19901 tablet.cc:2369] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:42:19.551788 19767 tablet.cc:2369] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:19.576381 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.572717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.97.195:41707)
0925 16:42:19.572882 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.572886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.572901 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:42:19.573069 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.573072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.573127 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.573241 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:19.573515 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.574061 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.574926 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.575150 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.575258 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.575565 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:19.575975 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:19.576107 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.576148 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.576204 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:42:19.582059 19854 raft_consensus.cc:1261] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 1 FOLLOWER]: Refusing update from remote peer 3f355744092a43efbefbed3e1149b711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:19.582059 19720 raft_consensus.cc:1261] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 38d470acc24540dcbb3e77f8b57d3d42 [term 1 FOLLOWER]: Refusing update from remote peer 3f355744092a43efbefbed3e1149b711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:19.582746 20058 consensus_queue.cc:1035] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d470acc24540dcbb3e77f8b57d3d42" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:19.582955 20061 consensus_queue.cc:1035] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d25e01782546ada3eb8e796842b114" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:19.591089 20066 mvcc.cc:205] Tried to move back new op lower bound from 7333292194120962048 to 7333292193833918464. Current Snapshot: MvccSnapshot[applied={T|T < 7333292194120962048}]
I20260925 16:42:19.592186 20065 mvcc.cc:205] Tried to move back new op lower bound from 7333292194120962048 to 7333292193833918464. Current Snapshot: MvccSnapshot[applied={T|T < 7333292194120962048}]
I20260925 16:42:19.599416 19573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:19.599587 19573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:19.601835 19573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbbf95507a114c7fb6b8a46c8c028141: Sending DeleteTablet for 3 replicas of tablet 2c98c76f9cf74ed2b05a60624dcd3c5f
I20260925 16:42:19.602645 19700 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:19 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:50476
I20260925 16:42:19.602775 19834 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:19 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:60344
I20260925 16:42:19.603127 19968 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:19 UTC) from {username='kudu', principal='kudu/127.0.97.254@KRBTEST.COM'} at 127.0.0.1:42670
I20260925 16:42:19.603442 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19637
I20260925 16:42:19.605602 20075 tablet_replica.cc:330] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: stopping tablet replica
I20260925 16:42:19.605661 20076 tablet_replica.cc:330] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: stopping tablet replica
I20260925 16:42:19.605989 20076 raft_consensus.cc:2229] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:19.606033 20075 raft_consensus.cc:2229] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:19.606369 20076 raft_consensus.cc:2258] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:19.606413 20075 raft_consensus.cc:2258] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:19.608003 20076 ts_tablet_manager.cc:1905] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:19.608003 20075 ts_tablet_manager.cc:1905] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:19.610707 20075 ts_tablet_manager.cc:1918] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:19.610707 20076 ts_tablet_manager.cc:1918] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:19.610803 20076 log.cc:1196] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/2c98c76f9cf74ed2b05a60624dcd3c5f
I20260925 16:42:19.610909 20075 log.cc:1196] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/2c98c76f9cf74ed2b05a60624dcd3c5f
W20260925 16:42:19.611172 19559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:45847 (error 108)
I20260925 16:42:19.611222 20076 ts_tablet_manager.cc:1939] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 3f355744092a43efbefbed3e1149b711: Deleting consensus metadata
I20260925 16:42:19.611343 20075 ts_tablet_manager.cc:1939] T 2c98c76f9cf74ed2b05a60624dcd3c5f P 22d25e01782546ada3eb8e796842b114: Deleting consensus metadata
I20260925 16:42:19.612040 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19771
I20260925 16:42:19.613111 19561 catalog_manager.cc:4841] TS 3f355744092a43efbefbed3e1149b711 (127.0.97.195:41707): tablet 2c98c76f9cf74ed2b05a60624dcd3c5f (table test-table [id=dcbbfa7e00574f919fa7360124e29a2d]) successfully deleted
I20260925 16:42:19.615339 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.614987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:45847 (local address 127.0.0.1:50486)
0925 16:42:19.615152 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.615236 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 16:42:19.615577 19559 catalog_manager.cc:4584] TS 38d470acc24540dcbb3e77f8b57d3d42 (127.0.97.193:45847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c98c76f9cf74ed2b05a60624dcd3c5f: Network error: Client connection negotiation failed: client connection to 127.0.97.193:45847: connect: Connection refused (error 111)
I20260925 16:42:19.617800 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19905
I20260925 16:42:19.618868 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.617961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:41435 (local address 127.0.0.1:60352)
0925 16:42:19.618762 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:19.618805 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.194:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745}
I20260925 16:42:19.623271 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 19543
2026-09-25T16:42:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2433 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T16:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:19Z Disabled control of system clock
I20260925 16:42:19.653167 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36841
--webserver_interface=127.0.97.254
--webserver_port=0
--builtin_ntp_servers=127.0.97.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:36841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:19.760893 20081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:19.761147 20081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:19.761180 20081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:19.765026 20081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:19.765082 20081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:19.765098 20081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:19.765115 20081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:19.765129 20081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:19.769816 20081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:36841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:36841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:19.770814 20081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:19.772179 20081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:19.780694 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:19.780704 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:19.780884 20081 server_base.cc:1034] running on GCE node
W20260925 16:42:19.781033 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:19.781615 20081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:19.782717 20081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:19.783910 20081 hybrid_clock.cc:648] HybridClock initialized: now 1790354539783897 us; error 26 us; skew 500 ppm
I20260925 16:42:19.785786 20081 webserver.cc:460] Webserver started at http://127.0.97.254:42983/ using document root <none> and password file <none>
I20260925 16:42:19.786353 20081 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:19.786448 20081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:19.786679 20081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:19.788702 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "802bea9706e64c10bed6c61768896a11"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "33b68f2efeea50337e9020c0f2465529"
server_key_iv: "add77e0f5d881c5e73c00e43c1f3337e"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.789256 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "802bea9706e64c10bed6c61768896a11"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "33b68f2efeea50337e9020c0f2465529"
server_key_iv: "add77e0f5d881c5e73c00e43c1f3337e"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.792770 20081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:42:19.795210 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:19.796396 20081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:42:19.796535 20081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "802bea9706e64c10bed6c61768896a11"
format_stamp: "Formatted at 2026-09-25 16:42:19 on dist-test-slave-kxl8"
server_key: "33b68f2efeea50337e9020c0f2465529"
server_key_iv: "add77e0f5d881c5e73c00e43c1f3337e"
server_key_version: "encryptionkey@0"
I20260925 16:42:19.796651 20081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:19.822988 20081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:19.823722 20081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:19.823935 20081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:19.830767 20081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:36841
I20260925 16:42:19.830796 20147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:36841 every 8 connection(s)
I20260925 16:42:19.831689 20081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:19.834632 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:19.838557 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20081
I20260925 16:42:19.838657 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:19.838970 391 external_mini_cluster.cc:1428] Setting key 199ca504d4c07a1954ba0aead86c7f03
I20260925 16:42:19.840251 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: Bootstrap starting.
I20260925 16:42:19.842617 20148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:19.843402 20148 log.cc:824] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:19.845364 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: No bootstrap required, opened a new log
I20260925 16:42:19.848698 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:19.840397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.0.97.254:36841)
0925 16:42:19.841014 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:19.841028 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:19.841062 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:42:19.841739 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:19.841757 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:19.842313 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:19.843110 (+ 797us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:19.843447 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.844440 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.845632 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:19.846095 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:19.846211 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:19.847029 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:19.847034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:19.847045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:19.847417 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:19.847472 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:19.847491 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:19.847568 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:19.847731 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 16:42:19.848024 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260925 16:42:19.849254 20148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } }
I20260925 16:42:19.849537 20148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:19.849630 20148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802bea9706e64c10bed6c61768896a11, State: Initialized, Role: FOLLOWER
I20260925 16:42:19.850068 20148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } }
I20260925 16:42:19.850214 20148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:19.850302 20148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:19.850397 20148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:19.851387 20148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } }
I20260925 16:42:19.851713 20148 leader_election.cc:302] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 802bea9706e64c10bed6c61768896a11; no voters:
I20260925 16:42:19.852002 20148 leader_election.cc:288] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:19.852115 20153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:19.852376 20153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [term 1 LEADER]: Becoming Leader. State: Replica: 802bea9706e64c10bed6c61768896a11, State: Running, Role: LEADER
I20260925 16:42:19.852717 20153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } }
I20260925 16:42:19.853123 20148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:19.854341 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 802bea9706e64c10bed6c61768896a11. Latest consensus state: current_term: 1 leader_uuid: "802bea9706e64c10bed6c61768896a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } } }
I20260925 16:42:19.854597 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "802bea9706e64c10bed6c61768896a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802bea9706e64c10bed6c61768896a11" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 36841 } } }
I20260925 16:42:19.855125 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:19.855149 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:19.855690 20168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:19.858469 20168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:19.863642 20168 catalog_manager.cc:1347] Generated new cluster ID: cb289571fdd94e97b04b7ac05da02a54
I20260925 16:42:19.863704 20168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:19.870985 20168 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:19.872184 20168 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:19.882376 20168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: Generated new TSK 0
I20260925 16:42:19.883188 20168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:19.885674 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.193:0
--local_ip_for_outbound_sockets=127.0.97.193
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--builtin_ntp_servers=127.0.97.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:42:19.999004 20172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:19.999290 20172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:19.999382 20172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:20.004237 20172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:20.004304 20172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:20.004361 20172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.193
I20260925 16:42:20.008910 20172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.193
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.193
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:20.009913 20172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:20.011353 20172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:20.020835 20172 server_base.cc:1034] running on GCE node
W20260925 16:42:20.020910 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:20.021034 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:20.020838 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:20.021540 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:20.022223 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:20.023416 20172 hybrid_clock.cc:648] HybridClock initialized: now 1790354540023408 us; error 30 us; skew 500 ppm
I20260925 16:42:20.025354 20172 webserver.cc:460] Webserver started at http://127.0.97.193:46367/ using document root <none> and password file <none>
I20260925 16:42:20.025931 20172 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:20.026026 20172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:20.026255 20172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:20.028280 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data/instance:
uuid: "93d617c76349471fbedeb7961632cee8"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "93b5e759196b3269f435400572d6e77b"
server_key_iv: "98f993efe84b479df81217e49272d91b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.028823 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal/instance:
uuid: "93d617c76349471fbedeb7961632cee8"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "93b5e759196b3269f435400572d6e77b"
server_key_iv: "98f993efe84b479df81217e49272d91b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.032840 20172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:42:20.035220 20186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.036388 20172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:42:20.036523 20172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal
uuid: "93d617c76349471fbedeb7961632cee8"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "93b5e759196b3269f435400572d6e77b"
server_key_iv: "98f993efe84b479df81217e49272d91b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.036634 20172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:20.046550 20172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:20.047158 20172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:20.047343 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:20.047895 20172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:20.048768 20172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:20.048840 20172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.048923 20172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:20.048972 20172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.057650 20172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.193:42895
I20260925 16:42:20.057708 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.193:42895 every 8 connection(s)
I20260925 16:42:20.058511 20172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/data/info.pb
I20260925 16:42:20.061599 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20172
I20260925 16:42:20.061730 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-0/wal/instance
I20260925 16:42:20.062014 391 external_mini_cluster.cc:1428] Setting key b99fcd7333411843de1f6a2f58fccd51
I20260925 16:42:20.064114 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.194:0
--local_ip_for_outbound_sockets=127.0.97.194
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--builtin_ntp_servers=127.0.97.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:20.068361 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.060251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.193:40783 (local address 127.0.97.254:36841)
0925 16:42:20.060384 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.060388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.061479 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 16:42:20.063041 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.063043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.063084 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.063155 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:20.064273 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.064842 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.066211 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.066557 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.066645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.067909 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:20.067911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:20.067914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:20.067926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:20.067944 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:20.067952 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:20.068012 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.068206 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.068255 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:20.069655 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.060657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36841 (local address 127.0.97.193:40783)
0925 16:42:20.061317 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.061373 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.062825 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.063343 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.063350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.063435 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:20.064109 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.064131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.064965 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.064972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.066092 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.066101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.066713 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.066720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.066952 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.067614 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:20.067618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:20.067673 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:20.067759 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:20.068041 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:20.068052 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:20.068061 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.068195 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.068528 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260925 16:42:20.070891 20300 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36841
I20260925 16:42:20.071128 20300 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:20.071637 20300 heartbeater.cc:511] Master 127.0.97.254:36841 requested a full tablet report, sending...
I20260925 16:42:20.073258 20112 ts_manager.cc:194] Registered new tserver with Master: 93d617c76349471fbedeb7961632cee8 (127.0.97.193:42895)
I20260925 16:42:20.075032 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.193:40783
W20260925 16:42:20.174569 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:20.174813 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:20.174850 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:20.178763 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:20.178813 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:20.178872 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.194
I20260925 16:42:20.183491 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.194
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.194
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:20.184442 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:20.185729 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:20.195176 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:20.195184 20309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:20.195230 20303 server_base.cc:1034] running on GCE node
W20260925 16:42:20.195194 20311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:20.195868 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:20.196516 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:20.197726 20303 hybrid_clock.cc:648] HybridClock initialized: now 1790354540197695 us; error 44 us; skew 500 ppm
I20260925 16:42:20.199790 20303 webserver.cc:460] Webserver started at http://127.0.97.194:34415/ using document root <none> and password file <none>
I20260925 16:42:20.200423 20303 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:20.200512 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:20.200723 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:20.202876 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data/instance:
uuid: "3b7a1d162dfe4a6c910231495bcdf23c"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "2d08ce68ef244fb33a42a1f25aae2520"
server_key_iv: "af7516270db0ecbcbad5e0f84b15530a"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.203429 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal/instance:
uuid: "3b7a1d162dfe4a6c910231495bcdf23c"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "2d08ce68ef244fb33a42a1f25aae2520"
server_key_iv: "af7516270db0ecbcbad5e0f84b15530a"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.207026 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 16:42:20.209465 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.210700 20303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:20.210834 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal
uuid: "3b7a1d162dfe4a6c910231495bcdf23c"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "2d08ce68ef244fb33a42a1f25aae2520"
server_key_iv: "af7516270db0ecbcbad5e0f84b15530a"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.210945 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:20.231072 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:20.231770 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:20.231959 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:20.232547 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:20.233531 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:20.233600 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.233671 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:20.233723 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.242610 20303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.194:34541
I20260925 16:42:20.242645 20430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.194:34541 every 8 connection(s)
I20260925 16:42:20.243611 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/data/info.pb
I20260925 16:42:20.250741 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20303
I20260925 16:42:20.250870 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal/instance
I20260925 16:42:20.251200 391 external_mini_cluster.cc:1428] Setting key 0722e442c50e659910688bd870840f0a
I20260925 16:42:20.253398 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.245461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:40915 (local address 127.0.97.254:36841)
0925 16:42:20.245602 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.245606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.246771 (+ 1165us) server_negotiation.cc:407] Connection header received
0925 16:42:20.248150 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.248153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.248208 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.248291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:20.249169 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.249719 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.251098 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.251394 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.251498 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.252800 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:20.252810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:20.252813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:20.252824 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:20.252842 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:20.252850 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:20.252957 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.253131 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.253209 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:42:20.253474 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.195:0
--local_ip_for_outbound_sockets=127.0.97.195
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--builtin_ntp_servers=127.0.97.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:42:20.254474 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.245867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36841 (local address 127.0.97.194:40915)
0925 16:42:20.246617 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.246658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.247942 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.248469 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.248476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.248558 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:20.249023 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.249035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.249856 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.249860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.250957 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.250969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.251574 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.251579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.251746 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.252491 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:20.252495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:20.252550 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:20.252633 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:20.252963 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:20.252973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:20.252979 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.253111 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.253442 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":134,"threads_started":1}
I20260925 16:42:20.256012 20431 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36841
I20260925 16:42:20.256251 20431 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:20.256750 20431 heartbeater.cc:511] Master 127.0.97.254:36841 requested a full tablet report, sending...
I20260925 16:42:20.257856 20112 ts_manager.cc:194] Registered new tserver with Master: 3b7a1d162dfe4a6c910231495bcdf23c (127.0.97.194:34541)
I20260925 16:42:20.258733 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.194:40915
W20260925 16:42:20.364022 20434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:20.364269 20434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:20.364305 20434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:20.368198 20434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:20.368249 20434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:20.368301 20434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.195
I20260925 16:42:20.372910 20434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.195
--webserver_port=0
--tserver_master_addrs=127.0.97.254:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.195
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:20.373898 20434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:20.375157 20434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:20.384779 20439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:20.384768 20440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:20.384914 20434 server_base.cc:1034] running on GCE node
W20260925 16:42:20.384763 20442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:20.385510 20434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:20.386159 20434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:20.387333 20434 hybrid_clock.cc:648] HybridClock initialized: now 1790354540387296 us; error 47 us; skew 500 ppm
I20260925 16:42:20.393728 20434 webserver.cc:460] Webserver started at http://127.0.97.195:34439/ using document root <none> and password file <none>
I20260925 16:42:20.394320 20434 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:20.394415 20434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:20.394666 20434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:20.396720 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data/instance:
uuid: "bc98a0dadee14d07a533051da44811fe"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "25575653779120075b8ba962bb42da7f"
server_key_iv: "66fd2a70c3fc992c576e9f086697e08b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.397320 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal/instance:
uuid: "bc98a0dadee14d07a533051da44811fe"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "25575653779120075b8ba962bb42da7f"
server_key_iv: "66fd2a70c3fc992c576e9f086697e08b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.401052 20434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:42:20.403414 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.404544 20434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:20.404675 20434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal
uuid: "bc98a0dadee14d07a533051da44811fe"
format_stamp: "Formatted at 2026-09-25 16:42:20 on dist-test-slave-kxl8"
server_key: "25575653779120075b8ba962bb42da7f"
server_key_iv: "66fd2a70c3fc992c576e9f086697e08b"
server_key_version: "encryptionkey@0"
I20260925 16:42:20.404783 20434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:20.420667 20434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:20.421380 20434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:20.421573 20434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:20.422142 20434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:42:20.423058 20434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:42:20.423130 20434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.423198 20434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:42:20.423251 20434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:20.431632 20434 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.195:38695
I20260925 16:42:20.431663 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.195:38695 every 8 connection(s)
I20260925 16:42:20.432489 20434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/data/info.pb
I20260925 16:42:20.440112 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20434
I20260925 16:42:20.440197 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal/instance
I20260925 16:42:20.440496 391 external_mini_cluster.cc:1428] Setting key 0f7d7c795dbb0a2d71a183489168f055
I20260925 16:42:20.441339 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.434167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.195:59337 (local address 127.0.97.254:36841)
0925 16:42:20.434300 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.434303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.435236 (+ 933us) server_negotiation.cc:407] Connection header received
0925 16:42:20.436570 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.436573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.436620 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.436693 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:20.437624 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.438142 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.439229 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.439437 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.439513 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.440748 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:20.440752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:20.440755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:20.440768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:20.440790 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:20.440803 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:20.440878 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.441094 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.441153 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:42:20.442245 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.434494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:36841 (local address 127.0.97.195:59337)
0925 16:42:20.435107 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.435145 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.436380 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.436847 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.436854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.436933 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:20.437497 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.437511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.438241 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.438243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.439128 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.439136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.439560 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.439562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.439701 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.440459 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:42:20.440460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:20.440499 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:42:20.440596 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:20.440905 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:20.440913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:20.440920 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.441089 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.441409 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260925 16:42:20.443244 20562 heartbeater.cc:347] Connected to a master server at 127.0.97.254:36841
I20260925 16:42:20.443485 20562 heartbeater.cc:464] Registering TS with master...
I20260925 16:42:20.443969 20562 heartbeater.cc:511] Master 127.0.97.254:36841 requested a full tablet report, sending...
I20260925 16:42:20.444908 20112 ts_manager.cc:194] Registered new tserver with Master: bc98a0dadee14d07a533051da44811fe (127.0.97.195:38695)
I20260925 16:42:20.445518 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.195:59337
I20260925 16:42:20.454715 391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:42:20.460641 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.456797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.0.97.254:36841)
0925 16:42:20.456949 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.456954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.457100 (+ 146us) server_negotiation.cc:407] Connection header received
0925 16:42:20.457249 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.457252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.457353 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.457446 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:20.457743 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.458263 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.459108 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.459328 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.459450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.460195 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:20.460198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:20.460201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:20.460219 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:20.460234 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:20.460242 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:20.460321 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.460439 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.460493 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:42:20.463515 20112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:42:20.466559 20112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:42:20.480769 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.475504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.0.97.195:38695)
0925 16:42:20.476126 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.476137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.476161 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:42:20.476235 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.476243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.476421 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.476567 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:20.476767 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.477710 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.479557 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480090 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.480216 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.480432 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.480501 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.480593 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":128,"threads_started":1}
I20260925 16:42:20.480760 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.474889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:38695 (local address 127.0.0.1:55444)
0925 16:42:20.476032 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.476046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.476126 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.476552 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.476556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.476574 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:20.476678 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.476684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.477827 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.477830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.479414 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.479420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480216 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.480220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.480327 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.480362 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.480462 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.480593 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":68,"thread_start_us":107,"threads_started":1}
I20260925 16:42:20.480898 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.474632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.194:34541 (local address 127.0.0.1:59548)
0925 16:42:20.475046 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.475100 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.475219 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.476303 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.476310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.476339 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:20.476507 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.476514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.477969 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.477971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.479409 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.479420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480420 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.480437 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.480533 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.480555 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.480610 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.480685 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":107,"spinlock_wait_cycles":2304,"thread_start_us":130,"threads_started":1}
I20260925 16:42:20.481230 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.475576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.193:42895)
0925 16:42:20.475959 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.475970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.476506 (+ 536us) server_negotiation.cc:407] Connection header received
0925 16:42:20.476590 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.476598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.476785 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.476932 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:20.477158 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.478276 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.479481 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480279 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.480397 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.480578 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.480984 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.481080 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":148,"threads_started":1}
I20260925 16:42:20.481581 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.474809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.0.97.194:34541)
0925 16:42:20.475783 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.475797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.475823 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:42:20.475902 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.475908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.476123 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.476296 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:20.476636 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.477858 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.479523 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480279 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.481048 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.481221 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.481299 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.481408 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":173,"threads_started":1}
I20260925 16:42:20.481779 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.475425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42895 (local address 127.0.0.1:54740)
0925 16:42:20.476304 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.476315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.476403 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.476931 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.476934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.476953 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:20.477028 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.477034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.478392 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.478395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.479336 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.479345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.480636 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.480639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.480713 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.480733 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.480797 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.480857 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":33,"spinlock_wait_cycles":3712,"thread_start_us":85,"threads_started":1}
I20260925 16:42:20.484198 20365 tablet_service.cc:1467] Processing CreateTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c (DEFAULT_TABLE table=test-table [id=e891694d42f0499c88fac8ac1f7436c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:20.484294 20496 tablet_service.cc:1467] Processing CreateTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c (DEFAULT_TABLE table=test-table [id=e891694d42f0499c88fac8ac1f7436c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:20.484483 20234 tablet_service.cc:1467] Processing CreateTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c (DEFAULT_TABLE table=test-table [id=e891694d42f0499c88fac8ac1f7436c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:42:20.485235 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ff2cc821b4f1ca5494df34d4d9b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:20.485324 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ff2cc821b4f1ca5494df34d4d9b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:20.485324 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ff2cc821b4f1ca5494df34d4d9b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:20.491299 20579 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Bootstrap starting.
I20260925 16:42:20.492548 20581 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Bootstrap starting.
I20260925 16:42:20.493580 20580 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Bootstrap starting.
I20260925 16:42:20.493760 20579 tablet_bootstrap.cc:654] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:20.494659 20581 tablet_bootstrap.cc:654] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:20.494638 20579 log.cc:824] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:20.495384 20581 log.cc:824] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:20.495539 20580 tablet_bootstrap.cc:654] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:20.496328 20580 log.cc:824] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:20.497418 20579 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: No bootstrap required, opened a new log
I20260925 16:42:20.497424 20581 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: No bootstrap required, opened a new log
I20260925 16:42:20.497619 20579 ts_tablet_manager.cc:1397] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:42:20.497619 20581 ts_tablet_manager.cc:1397] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:42:20.498214 20580 tablet_bootstrap.cc:492] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: No bootstrap required, opened a new log
I20260925 16:42:20.498385 20580 ts_tablet_manager.cc:1397] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:42:20.501215 20579 raft_consensus.cc:348] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.501545 20579 raft_consensus.cc:374] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:20.501622 20579 raft_consensus.cc:729] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7a1d162dfe4a6c910231495bcdf23c, State: Initialized, Role: FOLLOWER
I20260925 16:42:20.501951 20581 raft_consensus.cc:348] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.502084 20579 consensus_queue.cc:260] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.502202 20581 raft_consensus.cc:374] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:20.502449 20581 raft_consensus.cc:729] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d617c76349471fbedeb7961632cee8, State: Initialized, Role: FOLLOWER
I20260925 16:42:20.502926 20581 consensus_queue.cc:260] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.503153 20579 ts_tablet_manager.cc:1428] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:42:20.503590 20431 heartbeater.cc:503] Master 127.0.97.254:36841 was elected leader, sending a full tablet report...
I20260925 16:42:20.503939 20581 ts_tablet_manager.cc:1428] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:42:20.502544 20580 raft_consensus.cc:348] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.504245 20300 heartbeater.cc:503] Master 127.0.97.254:36841 was elected leader, sending a full tablet report...
I20260925 16:42:20.504297 20580 raft_consensus.cc:374] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:20.504381 20580 raft_consensus.cc:729] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc98a0dadee14d07a533051da44811fe, State: Initialized, Role: FOLLOWER
I20260925 16:42:20.504922 20580 consensus_queue.cc:260] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.506167 20580 ts_tablet_manager.cc:1428] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:42:20.506552 20562 heartbeater.cc:503] Master 127.0.97.254:36841 was elected leader, sending a full tablet report...
W20260925 16:42:20.559734 20301 tablet.cc:2369] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:42:20.683494 20563 tablet.cc:2369] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:42:20.744726 20432 tablet.cc:2369] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:20.838178 20585 raft_consensus.cc:482] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:42:20.838397 20585 raft_consensus.cc:504] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.839653 20585 leader_election.cc:288] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93d617c76349471fbedeb7961632cee8 (127.0.97.193:42895), bc98a0dadee14d07a533051da44811fe (127.0.97.195:38695)
I20260925 16:42:20.844223 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.840292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:58939 (local address 127.0.97.195:38695)
0925 16:42:20.840447 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.840454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.840470 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:42:20.840534 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.840536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.840599 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.840679 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:20.840931 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.841602 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843077 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.843668 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843791 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.843841 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.844039 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.844108 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:20.844249 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.839880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.195:38695 (local address 127.0.97.194:58939)
0925 16:42:20.840243 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.840259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.840363 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.840704 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.840707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.840724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:20.840836 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.840841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.841713 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.841720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.842933 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.842943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.843791 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.843885 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.843936 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.844023 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.844092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260925 16:42:20.844347 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.840012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.194:43297 (local address 127.0.97.193:42895)
0925 16:42:20.840139 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.840143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.840450 (+ 307us) server_negotiation.cc:407] Connection header received
0925 16:42:20.840653 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.840658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.840713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.840781 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:42:20.841136 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.841918 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843271 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.843661 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843794 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.843841 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.844120 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.844201 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:20.844687 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.839880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42895 (local address 127.0.97.194:43297)
0925 16:42:20.840355 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.840370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:20.840522 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:20.840819 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:20.840822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:20.840840 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:42:20.840968 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.840975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.842055 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.842058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.843158 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:20.843165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.843785 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.843789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:20.843953 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.843972 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:42:20.844502 (+ 530us) client_negotiation.cc:240] Negotiation successful
0925 16:42:20.844586 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":86,"thread_start_us":79,"threads_started":1}
I20260925 16:42:20.845324 20254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ff2cc821b4f1ca5494df34d4d9b5c" candidate_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d617c76349471fbedeb7961632cee8" is_pre_election: true
I20260925 16:42:20.845335 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ff2cc821b4f1ca5494df34d4d9b5c" candidate_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98a0dadee14d07a533051da44811fe" is_pre_election: true
I20260925 16:42:20.845629 20254 raft_consensus.cc:2454] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b7a1d162dfe4a6c910231495bcdf23c in term 0.
I20260925 16:42:20.845629 20516 raft_consensus.cc:2454] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b7a1d162dfe4a6c910231495bcdf23c in term 0.
I20260925 16:42:20.846135 20320 leader_election.cc:302] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b7a1d162dfe4a6c910231495bcdf23c, 93d617c76349471fbedeb7961632cee8, bc98a0dadee14d07a533051da44811fe; no voters:
I20260925 16:42:20.846410 20585 raft_consensus.cc:2781] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:42:20.846604 20585 raft_consensus.cc:482] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:42:20.846668 20585 raft_consensus.cc:3037] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:20.847757 20585 raft_consensus.cc:504] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.848183 20585 leader_election.cc:288] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [CANDIDATE]: Term 1 election: Requested vote from peers 93d617c76349471fbedeb7961632cee8 (127.0.97.193:42895), bc98a0dadee14d07a533051da44811fe (127.0.97.195:38695)
I20260925 16:42:20.848515 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ff2cc821b4f1ca5494df34d4d9b5c" candidate_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98a0dadee14d07a533051da44811fe"
I20260925 16:42:20.848506 20254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ff2cc821b4f1ca5494df34d4d9b5c" candidate_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d617c76349471fbedeb7961632cee8"
I20260925 16:42:20.848632 20254 raft_consensus.cc:3037] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:20.848632 20516 raft_consensus.cc:3037] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:20.849948 20254 raft_consensus.cc:2454] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7a1d162dfe4a6c910231495bcdf23c in term 1.
I20260925 16:42:20.849943 20516 raft_consensus.cc:2454] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7a1d162dfe4a6c910231495bcdf23c in term 1.
I20260925 16:42:20.850260 20318 leader_election.cc:302] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7a1d162dfe4a6c910231495bcdf23c, 93d617c76349471fbedeb7961632cee8; no voters:
I20260925 16:42:20.850448 20585 raft_consensus.cc:2781] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:20.850636 20585 raft_consensus.cc:686] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 1 LEADER]: Becoming Leader. State: Replica: 3b7a1d162dfe4a6c910231495bcdf23c, State: Running, Role: LEADER
I20260925 16:42:20.850958 20585 consensus_queue.cc:237] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } }
I20260925 16:42:20.854058 20111 catalog_manager.cc:5495] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b7a1d162dfe4a6c910231495bcdf23c (127.0.97.194). New cstate: current_term: 1 leader_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a1d162dfe4a6c910231495bcdf23c" member_type: VOTER last_known_addr { host: "127.0.97.194" port: 34541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 } health_report { overall_health: UNKNOWN } } }
I20260925 16:42:20.930500 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.926922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.97.194:34541)
0925 16:42:20.927066 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:20.927073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:20.927104 (+ 31us) server_negotiation.cc:407] Connection header received
0925 16:42:20.927190 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:20.927192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:20.927247 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:20.927359 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:20.927614 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.928121 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.928998 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:20.929218 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:20.929369 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:20.929688 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:20.930107 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:20.930243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:20.930304 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 16:42:20.930355 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:42:20.936471 20516 raft_consensus.cc:1261] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 1 FOLLOWER]: Refusing update from remote peer 3b7a1d162dfe4a6c910231495bcdf23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:20.936470 20254 raft_consensus.cc:1261] T 076ff2cc821b4f1ca5494df34d4d9b5c P 93d617c76349471fbedeb7961632cee8 [term 1 FOLLOWER]: Refusing update from remote peer 3b7a1d162dfe4a6c910231495bcdf23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:20.937189 20585 consensus_queue.cc:1035] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc98a0dadee14d07a533051da44811fe" member_type: VOTER last_known_addr { host: "127.0.97.195" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:20.937450 20590 consensus_queue.cc:1035] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d617c76349471fbedeb7961632cee8" member_type: VOTER last_known_addr { host: "127.0.97.193" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:20.947053 20595 mvcc.cc:205] Tried to move back new op lower bound from 7333292199668699136 to 7333292199328059392. Current Snapshot: MvccSnapshot[applied={T|T < 7333292199668699136}]
I20260925 16:42:20.947309 20594 mvcc.cc:205] Tried to move back new op lower bound from 7333292199668699136 to 7333292199328059392. Current Snapshot: MvccSnapshot[applied={T|T < 7333292199668699136}]
I20260925 16:42:20.954795 20112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:20.954981 20112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:42:20.957661 20112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 802bea9706e64c10bed6c61768896a11: Sending DeleteTablet for 3 replicas of tablet 076ff2cc821b4f1ca5494df34d4d9b5c
I20260925 16:42:20.958516 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:20 UTC) from {username='slave'} at 127.0.0.1:55444
I20260925 16:42:20.958518 20365 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:20 UTC) from {username='slave'} at 127.0.0.1:59548
I20260925 16:42:20.958606 20234 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ff2cc821b4f1ca5494df34d4d9b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:42:20 UTC) from {username='slave'} at 127.0.0.1:54740
I20260925 16:42:20.959069 20604 tablet_replica.cc:330] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: stopping tablet replica
I20260925 16:42:20.959450 20604 raft_consensus.cc:2229] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:42:20.959559 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20172
I20260925 16:42:20.959812 20604 raft_consensus.cc:2258] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:20.960179 20603 tablet_replica.cc:330] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: stopping tablet replica
I20260925 16:42:20.960520 20603 raft_consensus.cc:2229] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:42:20.960945 20603 raft_consensus.cc:2258] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:42:20.962445 20603 ts_tablet_manager.cc:1905] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:20.963516 20604 ts_tablet_manager.cc:1905] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:42:20.965494 20603 ts_tablet_manager.cc:1918] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:20.965591 20603 log.cc:1196] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-1/wal/wals/076ff2cc821b4f1ca5494df34d4d9b5c
I20260925 16:42:20.965546 20604 ts_tablet_manager.cc:1918] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:42:20.965659 20604 log.cc:1196] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Deleting WAL directory at /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790354288967115-391-0/minicluster-data/ts-2/wal/wals/076ff2cc821b4f1ca5494df34d4d9b5c
I20260925 16:42:20.966030 20604 ts_tablet_manager.cc:1939] T 076ff2cc821b4f1ca5494df34d4d9b5c P bc98a0dadee14d07a533051da44811fe: Deleting consensus metadata
I20260925 16:42:20.966115 20603 ts_tablet_manager.cc:1939] T 076ff2cc821b4f1ca5494df34d4d9b5c P 3b7a1d162dfe4a6c910231495bcdf23c: Deleting consensus metadata
I20260925 16:42:20.967278 20098 catalog_manager.cc:4841] TS bc98a0dadee14d07a533051da44811fe (127.0.97.195:38695): tablet 076ff2cc821b4f1ca5494df34d4d9b5c (table test-table [id=e891694d42f0499c88fac8ac1f7436c1]) successfully deleted
I20260925 16:42:20.967280 20096 catalog_manager.cc:4841] TS 3b7a1d162dfe4a6c910231495bcdf23c (127.0.97.194:34541): tablet 076ff2cc821b4f1ca5494df34d4d9b5c (table test-table [id=e891694d42f0499c88fac8ac1f7436c1]) successfully deleted
I20260925 16:42:20.970649 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20303
W20260925 16:42:20.971628 20096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.193:42895 (error 108)
I20260925 16:42:20.973034 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:20.972613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.193:42895 (local address 127.0.0.1:54746)
0925 16:42:20.972866 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:20.972939 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.193:42895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260925 16:42:20.973212 20096 catalog_manager.cc:4584] TS 93d617c76349471fbedeb7961632cee8 (127.0.97.193:42895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 076ff2cc821b4f1ca5494df34d4d9b5c: Network error: Client connection negotiation failed: client connection to 127.0.97.193:42895: connect: Connection refused (error 111)
I20260925 16:42:20.977190 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20434
I20260925 16:42:20.982563 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20081
2026-09-25T16:42:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1358 ms)
[----------] 6 tests from EncryptionPolicyTest (10713 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:21 dist-test-slave-kxl8 krb5kdc[20608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:21 dist-test-slave-kxl8 krb5kdc[20608](info): set up 2 sockets
Sep 25 16:42:21 dist-test-slave-kxl8 krb5kdc[20608](info): commencing operation
krb5kdc: starting...
W20260925 16:42:22.013430 391 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:22.132725 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:22.240473 20621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:22.240731 20621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:22.240773 20621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:22.244865 20621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:22.244935 20621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:22.244956 20621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:22.245002 20621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:22.245035 20621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:22.250407 20621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:22.251567 20621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:22.252593 20621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:22.260821 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:22.260819 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:22.260980 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:22.261008 20621 server_base.cc:1034] running on GCE node
I20260925 16:42:22.261899 20621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:22.262902 20621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:22.264055 20621 hybrid_clock.cc:648] HybridClock initialized: now 1790354542264022 us; error 45 us; skew 500 ppm
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:22.266786 20621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:22.267688 20621 webserver.cc:460] Webserver started at http://127.0.0.1:46071/ using document root <none> and password file <none>
I20260925 16:42:22.268350 20621 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:22.268450 20621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:22.268679 20621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:22.270736 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad"
format_stamp: "Formatted at 2026-09-25 16:42:22 on dist-test-slave-kxl8"
server_key: "c5c29a2d067bd97923532ef91a016662"
server_key_iv: "d54cdb164cb76bb4c91660ed076571ab"
server_key_version: "encryptionkey@0"
I20260925 16:42:22.271320 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad"
format_stamp: "Formatted at 2026-09-25 16:42:22 on dist-test-slave-kxl8"
server_key: "c5c29a2d067bd97923532ef91a016662"
server_key_iv: "d54cdb164cb76bb4c91660ed076571ab"
server_key_version: "encryptionkey@0"
I20260925 16:42:22.275049 20621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:42:22.277661 20636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:22.278860 20621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:22.279013 20621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad"
format_stamp: "Formatted at 2026-09-25 16:42:22 on dist-test-slave-kxl8"
server_key: "c5c29a2d067bd97923532ef91a016662"
server_key_iv: "d54cdb164cb76bb4c91660ed076571ab"
server_key_version: "encryptionkey@0"
I20260925 16:42:22.279134 20621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:22.315670 20621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:22.318768 20621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:22.318994 20621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:22.326162 20621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34707
I20260925 16:42:22.326177 20688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34707 every 8 connection(s)
I20260925 16:42:22.327142 20621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:22.328696 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20621
I20260925 16:42:22.328842 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:22.329159 391 external_mini_cluster.cc:1428] Setting key efe8b0072c51f353097904d3302b4c48
I20260925 16:42:22.330359 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:22.335901 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: Bootstrap starting.
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:42:22.338514 20689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:22.339401 20689 log.cc:824] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:22.341693 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: No bootstrap required, opened a new log
I20260925 16:42:22.343546 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:22.330740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58737 (local address 127.0.0.1:34707)
0925 16:42:22.331179 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:22.331194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:22.331230 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:22.331864 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:22.331879 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:22.332218 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:22.332548 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:22.333376 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:22.334432 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:22.335272 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:22.335726 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:22.335898 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:22.338708 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:22.338725 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:22.338738 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:22.338764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:22.340664 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:22.341216 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:22.341221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:22.341225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:22.341390 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:22.341730 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:22.341732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:22.341733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:22.342133 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:22.342283 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:22.342614 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 16:42:22.342846 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":146,"threads_started":1}
I20260925 16:42:22.345614 20689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260925 16:42:22.345882 20689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:22.345971 20689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d55d75ca2ca4bbe8303c2f2f3b765ad, State: Initialized, Role: FOLLOWER
I20260925 16:42:22.346441 20689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260925 16:42:22.346599 20689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:22.346657 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:22.346665 20689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:22.346761 20689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:22.347627 20689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260925 16:42:22.347994 20689 leader_election.cc:302] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d55d75ca2ca4bbe8303c2f2f3b765ad; no voters:
I20260925 16:42:22.348320 20689 leader_election.cc:288] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:22.348404 20698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:22.348723 20698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [term 1 LEADER]: Becoming Leader. State: Replica: 5d55d75ca2ca4bbe8303c2f2f3b765ad, State: Running, Role: LEADER
I20260925 16:42:22.349068 20698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260925 16:42:22.349694 20689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:22.350296 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260925 16:42:22.350432 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:22.350636 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d55d75ca2ca4bbe8303c2f2f3b765ad. Latest consensus state: current_term: 1 leader_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d55d75ca2ca4bbe8303c2f2f3b765ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260925 16:42:22.350710 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad [sys.catalog]: This master's current role is: LEADER
W20260925 16:42:22.354532 20714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:22.354609 20714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:22.354672 20706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:22.355635 20706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:22.358171 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:22.348708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53125 (local address 127.0.0.1:34707)
0925 16:42:22.348831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:22.348835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:22.349054 (+ 219us) server_negotiation.cc:407] Connection header received
0925 16:42:22.349172 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:22.349175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:22.349214 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:22.349323 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:22.350015 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:22.351023 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:22.352416 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:22.352737 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:22.352844 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:22.354474 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:22.354478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:22.354481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:22.354505 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:22.356244 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:22.356799 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:22.356802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:22.356804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:22.356889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:22.357255 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:22.357259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:22.357260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:22.357503 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:22.357649 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:22.357921 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 16:42:22.358019 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:42:22.361778 20706 catalog_manager.cc:1347] Generated new cluster ID: 1236d82f550b489f9465e6e780e8646b
I20260925 16:42:22.361851 20706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:22.376842 20706 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:22.377946 20706 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:22.384205 20706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d55d75ca2ca4bbe8303c2f2f3b765ad: Generated new TSK 0
I20260925 16:42:22.384871 20706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:22.391124 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20621
W20260925 16:42:22.396495 20606 connection.cc:535] client connection to 127.0.0.1:34707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34707): Connection reset by peer (error 104)
2026-09-25T16:42:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20720](info): set up 2 sockets
Sep 25 16:42:22 dist-test-slave-kxl8 krb5kdc[20720](info): commencing operation
krb5kdc: starting...
W20260925 16:42:23.445456 391 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:23 dist-test-slave-kxl8 krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:23.568852 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:23.674719 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:23.675027 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:23.675102 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:23.679100 20733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:23.679196 20733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:23.679236 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:23.679296 20733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:23.679333 20733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:23.684512 20733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:23.685580 20733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:23.686583 20733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:23.695024 20733 server_base.cc:1034] running on GCE node
W20260925 16:42:23.695003 20739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:23.695044 20738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:23.695003 20741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:23.695798 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:23.696878 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:23.698040 20733 hybrid_clock.cc:648] HybridClock initialized: now 1790354543698021 us; error 30 us; skew 500 ppm
Sep 25 16:42:23 dist-test-slave-kxl8 krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:23.700773 20733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:23.701761 20733 webserver.cc:460] Webserver started at http://127.0.0.1:40133/ using document root <none> and password file <none>
I20260925 16:42:23.702322 20733 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:23.702383 20733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:23.702610 20733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:23.704643 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "168ba89d1b6249588979014381c320cf"
format_stamp: "Formatted at 2026-09-25 16:42:23 on dist-test-slave-kxl8"
server_key: "7fe0fa3994e355ccda32b751b917fd3f"
server_key_iv: "e5bbb8ed91078fac63f2cb1dfa88652a"
server_key_version: "encryptionkey@0"
I20260925 16:42:23.705204 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "168ba89d1b6249588979014381c320cf"
format_stamp: "Formatted at 2026-09-25 16:42:23 on dist-test-slave-kxl8"
server_key: "7fe0fa3994e355ccda32b751b917fd3f"
server_key_iv: "e5bbb8ed91078fac63f2cb1dfa88652a"
server_key_version: "encryptionkey@0"
I20260925 16:42:23.708959 20733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:42:23.711659 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:23.712934 20733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:23.713068 20733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "168ba89d1b6249588979014381c320cf"
format_stamp: "Formatted at 2026-09-25 16:42:23 on dist-test-slave-kxl8"
server_key: "7fe0fa3994e355ccda32b751b917fd3f"
server_key_iv: "e5bbb8ed91078fac63f2cb1dfa88652a"
server_key_version: "encryptionkey@0"
I20260925 16:42:23.713181 20733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:23.727360 20733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:23.730302 20733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:23.730512 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:23.737677 20733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45917
I20260925 16:42:23.737725 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45917 every 8 connection(s)
I20260925 16:42:23.738644 20733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:23.741410 20801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:23.744771 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20733
I20260925 16:42:23.744940 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:23.745220 391 external_mini_cluster.cc:1428] Setting key 55cad013bec97fe6f0189d7b933dd715
I20260925 16:42:23.747278 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: Bootstrap starting.
I20260925 16:42:23.749500 20801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:23.750216 20801 log.cc:824] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:23.752122 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: No bootstrap required, opened a new log
I20260925 16:42:23.754431 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:23.746675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55239 (local address 127.0.0.1:45917)
0925 16:42:23.747108 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:23.747123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:23.747158 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:42:23.747818 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:23.747836 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:23.748178 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:23.748519 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:23.748922 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:23.750024 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:23.750970 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:23.751454 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:23.751634 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:23.752571 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:23.752575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:23.752588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:23.752967 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:23.753022 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:23.753044 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:23.753181 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:23.753385 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 16:42:23.753664 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260925 16:42:23.755913 20801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 16:42:23.756183 20801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:23.756276 20801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168ba89d1b6249588979014381c320cf, State: Initialized, Role: FOLLOWER
I20260925 16:42:23.756758 20801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 16:42:23.756914 20801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:23.756990 20801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:23.757087 20801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:23.757483 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:23.758284 20801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 16:42:23.758651 20801 leader_election.cc:302] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168ba89d1b6249588979014381c320cf; no voters:
I20260925 16:42:23.759016 20801 leader_election.cc:288] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:23.759090 20810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:23.759330 20810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [term 1 LEADER]: Becoming Leader. State: Replica: 168ba89d1b6249588979014381c320cf, State: Running, Role: LEADER
I20260925 16:42:23.759640 20810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 16:42:23.760370 20801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:23.760931 20810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168ba89d1b6249588979014381c320cf. Latest consensus state: current_term: 1 leader_uuid: "168ba89d1b6249588979014381c320cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260925 16:42:23.761047 20810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:23.761399 20811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168ba89d1b6249588979014381c320cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168ba89d1b6249588979014381c320cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260925 16:42:23.761492 20811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:23.764660 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:23.759937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36435 (local address 127.0.0.1:45917)
0925 16:42:23.760036 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:23.760040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:23.760056 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:42:23.760123 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:23.760126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:23.760168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:23.760252 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:23.760669 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:23.761539 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:23.762663 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:23.763005 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:23.763084 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:23.764132 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:23.764137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:23.764140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:23.764153 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:23.764180 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:23.764194 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:23.764285 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:23.764483 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:42:23.764521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:42:23.765444 20826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:23.765516 20826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:23.765605 20816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:23.766273 20816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:23.771759 20816 catalog_manager.cc:1347] Generated new cluster ID: ccde475e4aa34668b223641246008895
I20260925 16:42:23.771829 20816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:23.780550 20816 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:23.781910 20816 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:23.787683 20816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168ba89d1b6249588979014381c320cf: Generated new TSK 0
I20260925 16:42:23.788393 20816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:23.797924 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20733
W20260925 16:42:23.806339 20718 connection.cc:535] client connection to 127.0.0.1:45917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45917): Connection reset by peer (error 104)
2026-09-25T16:42:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:23 dist-test-slave-kxl8 krb5kdc[20832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:23 dist-test-slave-kxl8 krb5kdc[20832](info): set up 2 sockets
Sep 25 16:42:23 dist-test-slave-kxl8 krb5kdc[20832](info): commencing operation
krb5kdc: starting...
W20260925 16:42:24.849375 391 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:24 dist-test-slave-kxl8 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:24.970350 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:25.080426 20845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:25.080734 20845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:25.080817 20845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:25.084839 20845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:25.084945 20845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:25.084986 20845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:25.085047 20845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:25.085086 20845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:25.090443 20845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:25.091523 20845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:25.092536 20845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:25.100731 20850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:25.100740 20853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:25.100934 20845 server_base.cc:1034] running on GCE node
W20260925 16:42:25.100761 20851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:25.101625 20845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:25.102736 20845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:25.103899 20845 hybrid_clock.cc:648] HybridClock initialized: now 1790354545103882 us; error 30 us; skew 500 ppm
Sep 25 16:42:25 dist-test-slave-kxl8 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:25.106739 20845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:25.107579 20845 webserver.cc:460] Webserver started at http://127.0.0.1:40697/ using document root <none> and password file <none>
I20260925 16:42:25.108119 20845 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:25.108177 20845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:25.108397 20845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:25.110396 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "5b850e4c9cb94df999eb1412b9d7981c"
format_stamp: "Formatted at 2026-09-25 16:42:25 on dist-test-slave-kxl8"
server_key: "7d2db6c7f49241385bb65a631426c10f"
server_key_iv: "8cb345043e3a5d464dfc5a4bd171a126"
server_key_version: "encryptionkey@0"
I20260925 16:42:25.110945 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "5b850e4c9cb94df999eb1412b9d7981c"
format_stamp: "Formatted at 2026-09-25 16:42:25 on dist-test-slave-kxl8"
server_key: "7d2db6c7f49241385bb65a631426c10f"
server_key_iv: "8cb345043e3a5d464dfc5a4bd171a126"
server_key_version: "encryptionkey@0"
I20260925 16:42:25.114586 20845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:25.116815 20860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:25.118028 20845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:25.118160 20845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "5b850e4c9cb94df999eb1412b9d7981c"
format_stamp: "Formatted at 2026-09-25 16:42:25 on dist-test-slave-kxl8"
server_key: "7d2db6c7f49241385bb65a631426c10f"
server_key_iv: "8cb345043e3a5d464dfc5a4bd171a126"
server_key_version: "encryptionkey@0"
I20260925 16:42:25.118268 20845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:25.122659 20845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:25.125468 20845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:25.125654 20845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:25.132355 20845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38343
I20260925 16:42:25.132432 20912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38343 every 8 connection(s)
I20260925 16:42:25.133266 20845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:25.135946 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:25.136169 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20845
I20260925 16:42:25.136324 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:25.136593 391 external_mini_cluster.cc:1428] Setting key 57079ceddeb86b12719c70493e0ceb25
I20260925 16:42:25.141959 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:25.138133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54613 (local address 127.0.0.1:38343)
0925 16:42:25.138544 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:25.138557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:25.138590 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:42:25.139269 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:25.139286 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:25.139566 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:25.139831 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:25.140662 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:25.140667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:25.140700 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:25.140721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:25.140790 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:25.140824 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:25.140937 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:25.141054 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 16:42:25.141213 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260925 16:42:25.142782 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c: Bootstrap starting.
I20260925 16:42:25.144610 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:25.145365 20913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:25.146248 20913 log.cc:824] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:25.148276 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c: No bootstrap required, opened a new log
I20260925 16:42:25.148285 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:25.146807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37039 (local address 127.0.0.1:38343)
0925 16:42:25.146935 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:25.146939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:25.147074 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:42:25.147159 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:25.147162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:25.147192 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:25.147266 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:25.147940 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:25.147943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:25.147946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:25.147957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:25.147985 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:25.147997 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:25.148089 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:25.148150 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 16:42:25.148183 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:25.151610 20913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } }
I20260925 16:42:25.151813 20913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:25.151846 20913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b850e4c9cb94df999eb1412b9d7981c, State: Initialized, Role: FOLLOWER
I20260925 16:42:25.152240 20913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } }
I20260925 16:42:25.152359 20913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:25.152398 20913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:25.152467 20913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:25.153263 20913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } }
I20260925 16:42:25.153595 20913 leader_election.cc:302] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b850e4c9cb94df999eb1412b9d7981c; no voters:
I20260925 16:42:25.153832 20913 leader_election.cc:288] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:25.153975 20923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:25.154201 20923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [term 1 LEADER]: Becoming Leader. State: Replica: 5b850e4c9cb94df999eb1412b9d7981c, State: Running, Role: LEADER
I20260925 16:42:25.154518 20923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } }
I20260925 16:42:25.155020 20913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:25.156101 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b850e4c9cb94df999eb1412b9d7981c. Latest consensus state: current_term: 1 leader_uuid: "5b850e4c9cb94df999eb1412b9d7981c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } } }
I20260925 16:42:25.156225 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:25.156495 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b850e4c9cb94df999eb1412b9d7981c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b850e4c9cb94df999eb1412b9d7981c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38343 } } }
I20260925 16:42:25.156562 20932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:25.156590 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:25.159804 20932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:25.165620 20932 catalog_manager.cc:1347] Generated new cluster ID: 14bccc8f1a5844bd921adc7d0b707b12
I20260925 16:42:25.165683 20932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:25.174623 20932 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:25.174924 20932 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:25.183982 20932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b850e4c9cb94df999eb1412b9d7981c: Generated new TSK 0
I20260925 16:42:25.184757 20932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:25.194993 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20845
2026-09-25T16:42:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:25 dist-test-slave-kxl8 krb5kdc[20944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:25 dist-test-slave-kxl8 krb5kdc[20944](info): set up 2 sockets
Sep 25 16:42:25 dist-test-slave-kxl8 krb5kdc[20944](info): commencing operation
krb5kdc: starting...
W20260925 16:42:26.233587 391 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:26.352711 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:26.462167 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:26.462474 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:26.462548 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:26.466583 20957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:26.466666 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:26.466708 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:26.466763 20957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:26.466799 20957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:26.472134 20957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:26.473218 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:26.474287 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:26.482692 20957 server_base.cc:1034] running on GCE node
W20260925 16:42:26.482861 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:26.482903 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:26.482875 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:26.483541 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:26.484683 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:26.485865 20957 hybrid_clock.cc:648] HybridClock initialized: now 1790354546485830 us; error 51 us; skew 500 ppm
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:26.496856 20957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:26.497869 20957 webserver.cc:460] Webserver started at http://127.0.0.1:44143/ using document root <none> and password file <none>
I20260925 16:42:26.498458 20957 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:26.498519 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:26.498687 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:26.500669 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "fa5d1da774a34c089273b9c8374599b1"
format_stamp: "Formatted at 2026-09-25 16:42:26 on dist-test-slave-kxl8"
server_key: "b42718dfba00a4e00cbcf573f14356e8"
server_key_iv: "c2e07e69ce65fba5ce2162224ae27891"
server_key_version: "encryptionkey@0"
I20260925 16:42:26.501176 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "fa5d1da774a34c089273b9c8374599b1"
format_stamp: "Formatted at 2026-09-25 16:42:26 on dist-test-slave-kxl8"
server_key: "b42718dfba00a4e00cbcf573f14356e8"
server_key_iv: "c2e07e69ce65fba5ce2162224ae27891"
server_key_version: "encryptionkey@0"
I20260925 16:42:26.506400 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 16:42:26.510288 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:26.512564 20957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 16:42:26.512706 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "fa5d1da774a34c089273b9c8374599b1"
format_stamp: "Formatted at 2026-09-25 16:42:26 on dist-test-slave-kxl8"
server_key: "b42718dfba00a4e00cbcf573f14356e8"
server_key_iv: "c2e07e69ce65fba5ce2162224ae27891"
server_key_version: "encryptionkey@0"
I20260925 16:42:26.512833 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:26.528601 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:26.531504 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:26.531654 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:26.538448 20957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42097
I20260925 16:42:26.538470 21024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42097 every 8 connection(s)
I20260925 16:42:26.539453 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:26.541899 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 20957
I20260925 16:42:26.542068 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:26.542367 391 external_mini_cluster.cc:1428] Setting key 9e0d32f5902a8eca2696df59db697cc2
I20260925 16:42:26.542565 21025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:26.548887 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: Bootstrap starting.
I20260925 16:42:26.551298 21025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: Neither blocks nor log segments found. Creating new log.
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[20944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:42:26.552192 21025 log.cc:824] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:26.554364 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: No bootstrap required, opened a new log
I20260925 16:42:26.556779 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:26.543819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60619 (local address 127.0.0.1:42097)
0925 16:42:26.544281 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:26.544296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:26.544332 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:26.545253 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:26.545312 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:26.545730 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:26.546079 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:26.546870 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:26.547841 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:26.548694 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:26.549244 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:26.549445 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:26.552018 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:26.552023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:26.552035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:26.552062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:26.553896 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:26.554391 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:26.554395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:26.554400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:26.554552 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:26.554853 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:26.554856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:26.554857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:26.555228 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:26.555356 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:26.555716 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 16:42:26.556053 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":162,"threads_started":1}
I20260925 16:42:26.558282 21025 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } }
I20260925 16:42:26.558526 21025 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:26.558593 21025 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5d1da774a34c089273b9c8374599b1, State: Initialized, Role: FOLLOWER
I20260925 16:42:26.559013 21025 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } }
I20260925 16:42:26.559172 21025 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:26.559249 21025 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:26.559374 21025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:26.559634 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:26.560315 21025 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } }
I20260925 16:42:26.560654 21025 leader_election.cc:302] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5d1da774a34c089273b9c8374599b1; no voters:
I20260925 16:42:26.561013 21025 leader_election.cc:288] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:26.561126 21034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:26.561353 21034 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [term 1 LEADER]: Becoming Leader. State: Replica: fa5d1da774a34c089273b9c8374599b1, State: Running, Role: LEADER
I20260925 16:42:26.562186 21034 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } }
I20260925 16:42:26.562251 21025 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:26.563730 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5d1da774a34c089273b9c8374599b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } } }
I20260925 16:42:26.563853 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:26.564064 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5d1da774a34c089273b9c8374599b1. Latest consensus state: current_term: 1 leader_uuid: "fa5d1da774a34c089273b9c8374599b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5d1da774a34c089273b9c8374599b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42097 } } }
I20260925 16:42:26.564147 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1 [sys.catalog]: This master's current role is: LEADER
W20260925 16:42:26.567126 21050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:26.567217 21050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:26.567293 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:26.568130 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:26.573099 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:26.561760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52399 (local address 127.0.0.1:42097)
0925 16:42:26.561977 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:26.561981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:26.564466 (+ 2485us) server_negotiation.cc:407] Connection header received
0925 16:42:26.564693 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:26.564696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:26.564743 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:26.564790 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:26.565764 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:26.566533 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:26.567623 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:26.567848 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:26.567925 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:26.569544 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:26.569547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:26.569550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:26.569576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:26.571302 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:26.571826 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:26.571830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:26.571832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:26.571925 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:26.572268 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:26.572271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:26.572273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:26.572447 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:26.572585 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:26.572836 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 16:42:26.572949 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 16:42:26.574394 21049 catalog_manager.cc:1347] Generated new cluster ID: e2f1a8de6ce5427b94bbf45e800ec312
I20260925 16:42:26.574450 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:26.581622 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:26.582772 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:26.597718 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5d1da774a34c089273b9c8374599b1: Generated new TSK 0
I20260925 16:42:26.598599 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:26.602689 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 20957
2026-09-25T16:42:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[21056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[21056](info): set up 2 sockets
Sep 25 16:42:26 dist-test-slave-kxl8 krb5kdc[21056](info): commencing operation
krb5kdc: starting...
W20260925 16:42:27.637598 391 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:27 dist-test-slave-kxl8 krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:27.757984 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:27.866986 21069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:27.867307 21069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:27.867398 21069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:27.871503 21069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:27.871596 21069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:27.871634 21069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:27.871690 21069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:27.871727 21069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:27.877087 21069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:27.878186 21069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:27.879232 21069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:27.887733 21077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:27.887748 21075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:27.887748 21074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:27.888019 21069 server_base.cc:1034] running on GCE node
I20260925 16:42:27.888600 21069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:27.889740 21069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:27.890904 21069 hybrid_clock.cc:648] HybridClock initialized: now 1790354547890890 us; error 26 us; skew 500 ppm
Sep 25 16:42:27 dist-test-slave-kxl8 krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:27.893605 21069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:27.894443 21069 webserver.cc:460] Webserver started at http://127.0.0.1:36193/ using document root <none> and password file <none>
I20260925 16:42:27.894992 21069 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:27.895052 21069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:27.895296 21069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:27.897346 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "33b3a8a519274280b208059d0c5a708a"
format_stamp: "Formatted at 2026-09-25 16:42:27 on dist-test-slave-kxl8"
server_key: "2f5821c2c2e91bc167f4b017b0fca21d"
server_key_iv: "b438c3607e2dd2549c45ddf9d6375c90"
server_key_version: "encryptionkey@0"
I20260925 16:42:27.897918 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "33b3a8a519274280b208059d0c5a708a"
format_stamp: "Formatted at 2026-09-25 16:42:27 on dist-test-slave-kxl8"
server_key: "2f5821c2c2e91bc167f4b017b0fca21d"
server_key_iv: "b438c3607e2dd2549c45ddf9d6375c90"
server_key_version: "encryptionkey@0"
I20260925 16:42:27.901567 21069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:42:27.903945 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:27.905166 21069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:27.905349 21069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "33b3a8a519274280b208059d0c5a708a"
format_stamp: "Formatted at 2026-09-25 16:42:27 on dist-test-slave-kxl8"
server_key: "2f5821c2c2e91bc167f4b017b0fca21d"
server_key_iv: "b438c3607e2dd2549c45ddf9d6375c90"
server_key_version: "encryptionkey@0"
I20260925 16:42:27.905478 21069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:27.927363 21069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:27.930488 21069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:27.930697 21069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:27.938280 21069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42011
I20260925 16:42:27.938304 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42011 every 8 connection(s)
I20260925 16:42:27.939262 21069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:27.941967 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:27.944118 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21069
I20260925 16:42:27.944274 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:27.944530 391 external_mini_cluster.cc:1428] Setting key 05720be8e8c331eb4dde9a3d9ad68837
I20260925 16:42:27.947870 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a: Bootstrap starting.
I20260925 16:42:27.950220 21137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:27.950943 21137 log.cc:824] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:27.953171 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a: No bootstrap required, opened a new log
I20260925 16:42:27.953848 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:27.945937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34285 (local address 127.0.0.1:42011)
0925 16:42:27.946420 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:27.946436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:27.946471 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:42:27.947092 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:27.947109 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:27.947462 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:27.947799 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:27.948198 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:27.949146 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:27.950279 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:27.950806 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:27.950975 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:27.951805 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:27.951814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:27.951833 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:27.952335 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:27.952400 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:27.952434 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:27.952538 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:27.952677 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 16:42:27.952878 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":132,"threads_started":1}
I20260925 16:42:27.956693 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:27.956982 21137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260925 16:42:27.957249 21137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:27.957329 21137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b3a8a519274280b208059d0c5a708a, State: Initialized, Role: FOLLOWER
I20260925 16:42:27.957805 21137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260925 16:42:27.957968 21137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:27.958050 21137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:27.958240 21137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:27.959447 21137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260925 16:42:27.959839 21137 leader_election.cc:302] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b3a8a519274280b208059d0c5a708a; no voters:
I20260925 16:42:27.960223 21137 leader_election.cc:288] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:27.960260 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:27.960448 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [term 1 LEADER]: Becoming Leader. State: Replica: 33b3a8a519274280b208059d0c5a708a, State: Running, Role: LEADER
I20260925 16:42:27.960798 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260925 16:42:27.961499 21137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:27.962913 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b3a8a519274280b208059d0c5a708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } } }
I20260925 16:42:27.963045 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:27.963778 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:27.959030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46951 (local address 127.0.0.1:42011)
0925 16:42:27.959160 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:27.959165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:27.959437 (+ 272us) server_negotiation.cc:407] Connection header received
0925 16:42:27.959601 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:27.959604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:27.959654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:27.959713 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:27.960027 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:27.960683 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:27.961607 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:27.961861 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:27.961991 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:27.963065 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:27.963069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:27.963072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:27.963084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:27.963112 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:27.963125 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:27.963187 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:27.963565 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 16:42:27.963640 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:27.963969 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b3a8a519274280b208059d0c5a708a. Latest consensus state: current_term: 1 leader_uuid: "33b3a8a519274280b208059d0c5a708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3a8a519274280b208059d0c5a708a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } } }
I20260925 16:42:27.964071 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:27.964632 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:27.967332 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:27.973001 21157 catalog_manager.cc:1347] Generated new cluster ID: 43094e7e946b410c88f8e03c4f5a371b
I20260925 16:42:27.973075 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:27.978626 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:27.979771 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:27.989599 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b3a8a519274280b208059d0c5a708a: Generated new TSK 0
I20260925 16:42:27.990229 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:27.996757 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21069
2026-09-25T16:42:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:28 dist-test-slave-kxl8 krb5kdc[21168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:28 dist-test-slave-kxl8 krb5kdc[21168](info): set up 2 sockets
Sep 25 16:42:28 dist-test-slave-kxl8 krb5kdc[21168](info): commencing operation
krb5kdc: starting...
W20260925 16:42:29.053717 391 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:29 dist-test-slave-kxl8 krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:29.172267 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:29.281802 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:29.282105 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:29.282196 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:29.286249 21181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:29.286338 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:29.286374 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:29.286440 21181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:29.286478 21181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:29.291832 21181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:29.293038 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:29.294075 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:29.302382 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:29.302547 21181 server_base.cc:1034] running on GCE node
W20260925 16:42:29.302459 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:29.302385 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:29.303117 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:29.304237 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:29.305421 21181 hybrid_clock.cc:648] HybridClock initialized: now 1790354549305402 us; error 31 us; skew 500 ppm
Sep 25 16:42:29 dist-test-slave-kxl8 krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:29.308184 21181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:29.309046 21181 webserver.cc:460] Webserver started at http://127.0.0.1:34557/ using document root <none> and password file <none>
I20260925 16:42:29.309657 21181 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:29.309751 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:29.309986 21181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:29.312104 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "dbc4d76fb26d46aca7bc2121d82dc94c"
format_stamp: "Formatted at 2026-09-25 16:42:29 on dist-test-slave-kxl8"
server_key: "72efe40a6e84dd2bfe3d1732f01fa550"
server_key_iv: "c0ff65fafa5a654029ebee10b8e52267"
server_key_version: "encryptionkey@0"
I20260925 16:42:29.312677 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "dbc4d76fb26d46aca7bc2121d82dc94c"
format_stamp: "Formatted at 2026-09-25 16:42:29 on dist-test-slave-kxl8"
server_key: "72efe40a6e84dd2bfe3d1732f01fa550"
server_key_iv: "c0ff65fafa5a654029ebee10b8e52267"
server_key_version: "encryptionkey@0"
I20260925 16:42:29.316475 21181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 16:42:29.318902 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:29.320166 21181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:29.320308 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "dbc4d76fb26d46aca7bc2121d82dc94c"
format_stamp: "Formatted at 2026-09-25 16:42:29 on dist-test-slave-kxl8"
server_key: "72efe40a6e84dd2bfe3d1732f01fa550"
server_key_iv: "c0ff65fafa5a654029ebee10b8e52267"
server_key_version: "encryptionkey@0"
I20260925 16:42:29.320441 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:29.324986 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:29.327849 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:29.328047 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:29.334868 21181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35319
I20260925 16:42:29.334877 21248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35319 every 8 connection(s)
I20260925 16:42:29.335768 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:29.337886 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21181
I20260925 16:42:29.338049 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:29.338307 391 external_mini_cluster.cc:1428] Setting key 58c5ce2044aef701d4173d18da358f7a
I20260925 16:42:29.338676 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:29.343715 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:29.339685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47969 (local address 127.0.0.1:35319)
0925 16:42:29.340083 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:29.340101 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:29.340142 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:42:29.340888 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:29.340911 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:29.341262 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:29.341652 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:29.342356 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:29.342361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:29.342376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:29.342387 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:29.342460 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:29.342487 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:29.342628 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:29.342700 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 16:42:29.342843 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260925 16:42:29.345677 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c: Bootstrap starting.
I20260925 16:42:29.346592 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:29.348047 21249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:29.348943 21249 log.cc:824] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:29.350389 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:29.348626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33627 (local address 127.0.0.1:35319)
0925 16:42:29.348738 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:29.348741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:29.348975 (+ 234us) server_negotiation.cc:407] Connection header received
0925 16:42:29.349096 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:29.349099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:29.349136 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:29.349202 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:29.350013 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:29.350016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:29.350018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:29.350028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:29.350051 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:29.350060 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:29.350127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:29.350221 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 16:42:29.350250 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:42:29.351122 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c: No bootstrap required, opened a new log
I20260925 16:42:29.354660 21249 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } }
I20260925 16:42:29.354874 21249 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:29.354915 21249 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc4d76fb26d46aca7bc2121d82dc94c, State: Initialized, Role: FOLLOWER
I20260925 16:42:29.355265 21249 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } }
I20260925 16:42:29.355371 21249 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:29.355438 21249 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:29.355536 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:29.356345 21249 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } }
I20260925 16:42:29.356681 21249 leader_election.cc:302] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbc4d76fb26d46aca7bc2121d82dc94c; no voters:
I20260925 16:42:29.357060 21249 leader_election.cc:288] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:29.357127 21259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:29.357347 21259 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [term 1 LEADER]: Becoming Leader. State: Replica: dbc4d76fb26d46aca7bc2121d82dc94c, State: Running, Role: LEADER
I20260925 16:42:29.357642 21259 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } }
I20260925 16:42:29.358345 21249 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:29.358798 21259 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbc4d76fb26d46aca7bc2121d82dc94c. Latest consensus state: current_term: 1 leader_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } } }
I20260925 16:42:29.358906 21259 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:29.359102 21259 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc4d76fb26d46aca7bc2121d82dc94c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35319 } } }
I20260925 16:42:29.359261 21259 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:29.359267 21263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:29.362851 21263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:29.368597 21263 catalog_manager.cc:1347] Generated new cluster ID: 7f4d0454b0984725941742c0ea99a8e8
I20260925 16:42:29.368686 21263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:29.385577 21263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:29.385895 21263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:29.394541 21263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbc4d76fb26d46aca7bc2121d82dc94c: Generated new TSK 0
I20260925 16:42:29.395251 21263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:29.405759 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21181
2026-09-25T16:42:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:29 dist-test-slave-kxl8 krb5kdc[21280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:29 dist-test-slave-kxl8 krb5kdc[21280](info): set up 2 sockets
Sep 25 16:42:29 dist-test-slave-kxl8 krb5kdc[21280](info): commencing operation
krb5kdc: starting...
W20260925 16:42:30.437506 391 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:30.557057 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:30.668721 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:30.668996 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:30.669037 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:30.673127 21293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:30.673187 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:30.673205 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:30.673259 21293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:30.673326 21293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:30.678568 21293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:30.679559 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:30.680542 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:30.688812 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:30.688861 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:30.688985 21293 server_base.cc:1034] running on GCE node
W20260925 16:42:30.688813 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:30.689697 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:30.690800 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:30.691990 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790354550691980 us; error 30 us; skew 500 ppm
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:30.694684 21293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:30.695523 21293 webserver.cc:460] Webserver started at http://127.0.0.1:42813/ using document root <none> and password file <none>
I20260925 16:42:30.696076 21293 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:30.696166 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:30.696391 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:30.698442 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "a3590900deeb4c97b6483119eb6f2ba3"
format_stamp: "Formatted at 2026-09-25 16:42:30 on dist-test-slave-kxl8"
server_key: "593e88f1cc05b248f52e45412723eeed"
server_key_iv: "de36c60c00cf482e6199fa49ac683d28"
server_key_version: "encryptionkey@0"
I20260925 16:42:30.699007 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "a3590900deeb4c97b6483119eb6f2ba3"
format_stamp: "Formatted at 2026-09-25 16:42:30 on dist-test-slave-kxl8"
server_key: "593e88f1cc05b248f52e45412723eeed"
server_key_iv: "de36c60c00cf482e6199fa49ac683d28"
server_key_version: "encryptionkey@0"
I20260925 16:42:30.702615 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:30.704908 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:30.706184 21293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:30.706323 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "a3590900deeb4c97b6483119eb6f2ba3"
format_stamp: "Formatted at 2026-09-25 16:42:30 on dist-test-slave-kxl8"
server_key: "593e88f1cc05b248f52e45412723eeed"
server_key_iv: "de36c60c00cf482e6199fa49ac683d28"
server_key_version: "encryptionkey@0"
I20260925 16:42:30.706435 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:30.722745 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:30.725567 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:30.725764 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:30.732630 21293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41351
I20260925 16:42:30.732656 21360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41351 every 8 connection(s)
I20260925 16:42:30.733660 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:30.736207 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:30.741670 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3: Bootstrap starting.
I20260925 16:42:30.743049 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21293
I20260925 16:42:30.743197 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:30.743461 391 external_mini_cluster.cc:1428] Setting key 7314a2dbe62f9862df046f6b0d09c4c7
I20260925 16:42:30.744398 21361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:30.745266 21361 log.cc:824] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:30.747191 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3: No bootstrap required, opened a new log
I20260925 16:42:30.750610 21361 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260925 16:42:30.750844 21361 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:30.750923 21361 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3590900deeb4c97b6483119eb6f2ba3, State: Initialized, Role: FOLLOWER
I20260925 16:42:30.751410 21361 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260925 16:42:30.751559 21361 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:30.751646 21361 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:30.751754 21361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:42:30.752952 21361 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260925 16:42:30.753345 21361 leader_election.cc:302] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3590900deeb4c97b6483119eb6f2ba3; no voters:
I20260925 16:42:30.753690 21361 leader_election.cc:288] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:30.753801 21366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:30.754029 21366 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [term 1 LEADER]: Becoming Leader. State: Replica: a3590900deeb4c97b6483119eb6f2ba3, State: Running, Role: LEADER
I20260925 16:42:30.754415 21366 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260925 16:42:30.755617 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3590900deeb4c97b6483119eb6f2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260925 16:42:30.755748 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:30.756353 21361 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:30.756515 21370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:30.757514 21367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3590900deeb4c97b6483119eb6f2ba3. Latest consensus state: current_term: 1 leader_uuid: "a3590900deeb4c97b6483119eb6f2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3590900deeb4c97b6483119eb6f2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260925 16:42:30.757681 21367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:30.759094 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:30.744594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:57323 (local address 127.0.0.1:41351)
0925 16:42:30.745022 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:30.745041 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:30.745076 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:42:30.745966 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:30.745991 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:30.746361 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:30.746698 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:30.747535 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:30.748495 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:30.749394 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:30.749862 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:30.750038 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:30.752963 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:30.752969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:30.752982 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:30.753008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:30.755139 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:30.755846 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:30.755852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:30.755858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:30.755995 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:30.756385 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:30.756388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:30.756390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:30.756908 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:30.757136 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:30.757936 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 16:42:30.758171 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260925 16:42:30.759994 21370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:30.762307 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:30.766047 21370 catalog_manager.cc:1347] Generated new cluster ID: 7d733f6241a34fcfb133868338ba94cc
I20260925 16:42:30.766134 21370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:30.772400 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:30.764709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:41373 (local address 127.0.0.1:41351)
0925 16:42:30.764831 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:30.764836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:30.764849 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:42:30.764884 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:30.764886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:30.764927 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:30.765014 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:30.765882 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:30.766504 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:30.767312 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:30.767563 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:30.767668 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:30.769064 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:30.769067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:30.769070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:30.769094 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:30.770586 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:30.771062 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:30.771066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:30.771068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:30.771148 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:30.771498 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:30.771501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:30.771503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:30.771676 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:30.771845 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:30.772173 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 16:42:30.772274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260925 16:42:30.778537 21370 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:30.779431 21370 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:30.788501 21370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3590900deeb4c97b6483119eb6f2ba3: Generated new TSK 0
I20260925 16:42:30.789376 21370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:30.797137 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21293
W20260925 16:42:30.802543 21278 connection.cc:535] client connection to 127.0.0.1:41351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41351): Connection reset by peer (error 104)
2026-09-25T16:42:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21392](info): set up 2 sockets
Sep 25 16:42:30 dist-test-slave-kxl8 krb5kdc[21392](info): commencing operation
krb5kdc: starting...
W20260925 16:42:31.837519 391 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:31 dist-test-slave-kxl8 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:31.956766 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:32.067926 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:32.068176 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:32.068219 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:32.072132 21405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:32.072181 21405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:32.072201 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:32.072220 21405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:32.072234 21405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:32.077379 21405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:32.078362 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:32.079325 21405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:32.088124 21413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:32.088081 21410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:32.088143 21405 server_base.cc:1034] running on GCE node
W20260925 16:42:32.088093 21411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:32.088821 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:32.090000 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:32.091188 21405 hybrid_clock.cc:648] HybridClock initialized: now 1790354552091178 us; error 28 us; skew 500 ppm
Sep 25 16:42:32 dist-test-slave-kxl8 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:32.101575 21405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:32.102533 21405 webserver.cc:460] Webserver started at http://127.0.0.1:35883/ using document root <none> and password file <none>
I20260925 16:42:32.103094 21405 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:32.103183 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:32.103412 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:32.105505 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "ff4b8335f33b4515839678921d42c42e"
format_stamp: "Formatted at 2026-09-25 16:42:32 on dist-test-slave-kxl8"
server_key: "dcf15828bd43bc79ecfe0c26dec9054e"
server_key_iv: "bc6e4075e83a945eeda9342e797dcf5a"
server_key_version: "encryptionkey@0"
I20260925 16:42:32.106051 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "ff4b8335f33b4515839678921d42c42e"
format_stamp: "Formatted at 2026-09-25 16:42:32 on dist-test-slave-kxl8"
server_key: "dcf15828bd43bc79ecfe0c26dec9054e"
server_key_iv: "bc6e4075e83a945eeda9342e797dcf5a"
server_key_version: "encryptionkey@0"
I20260925 16:42:32.109743 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:32.112002 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:32.113245 21405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:32.113428 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "ff4b8335f33b4515839678921d42c42e"
format_stamp: "Formatted at 2026-09-25 16:42:32 on dist-test-slave-kxl8"
server_key: "dcf15828bd43bc79ecfe0c26dec9054e"
server_key_iv: "bc6e4075e83a945eeda9342e797dcf5a"
server_key_version: "encryptionkey@0"
I20260925 16:42:32.113557 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:32.127173 21405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:32.129916 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:32.130110 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:32.137140 21405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35541
I20260925 16:42:32.137164 21472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35541 every 8 connection(s)
I20260925 16:42:32.138079 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:32.140609 21473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:32.142908 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21405
I20260925 16:42:32.143055 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:32.143306 391 external_mini_cluster.cc:1428] Setting key f6db720297699653c6d4260cf4e32f64
I20260925 16:42:32.146732 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e: Bootstrap starting.
I20260925 16:42:32.149225 21473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:32.150223 21473 log.cc:824] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:32.152436 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e: No bootstrap required, opened a new log
I20260925 16:42:32.153136 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:32.144653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:53817 (local address 127.0.0.1:35541)
0925 16:42:32.145062 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:32.145080 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:32.145121 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:42:32.145966 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:32.145989 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:32.146396 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:32.146754 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:32.147145 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:32.148578 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:32.149520 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:32.150218 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:32.150376 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:32.151218 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:32.151224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:32.151239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:32.151722 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:32.151785 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:32.151813 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:32.151946 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:32.152112 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 16:42:32.152237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260925 16:42:32.155920 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:32.156582 21473 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260925 16:42:32.156780 21473 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:32.156839 21473 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4b8335f33b4515839678921d42c42e, State: Initialized, Role: FOLLOWER
I20260925 16:42:32.157335 21473 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260925 16:42:32.157469 21473 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:32.157554 21473 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:32.157647 21473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:32.158797 21473 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260925 16:42:32.159149 21473 leader_election.cc:302] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4b8335f33b4515839678921d42c42e; no voters:
I20260925 16:42:32.159454 21473 leader_election.cc:288] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:32.159566 21483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:32.159818 21483 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [term 1 LEADER]: Becoming Leader. State: Replica: ff4b8335f33b4515839678921d42c42e, State: Running, Role: LEADER
I20260925 16:42:32.160187 21483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260925 16:42:32.160815 21473 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:32.161262 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff4b8335f33b4515839678921d42c42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260925 16:42:32.161455 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:32.161847 21487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:32.161939 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4b8335f33b4515839678921d42c42e. Latest consensus state: current_term: 1 leader_uuid: "ff4b8335f33b4515839678921d42c42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4b8335f33b4515839678921d42c42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260925 16:42:32.162588 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:32.163187 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:32.158009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:52475 (local address 127.0.0.1:35541)
0925 16:42:32.158111 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:32.158115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:32.158269 (+ 154us) server_negotiation.cc:407] Connection header received
0925 16:42:32.158351 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:32.158354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:32.158395 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:32.158477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:32.158635 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:32.159410 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:32.160542 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:32.160863 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:32.160982 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:32.162429 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:32.162433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:32.162436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:32.162448 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:32.162483 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:32.162496 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:32.162556 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:32.163011 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 16:42:32.163048 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:42:32.164870 21487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:32.170440 21487 catalog_manager.cc:1347] Generated new cluster ID: 4b936590a5c94d678d73e5d5c3e99b08
I20260925 16:42:32.170521 21487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:32.180810 21487 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:32.181715 21487 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:32.188066 21487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff4b8335f33b4515839678921d42c42e: Generated new TSK 0
I20260925 16:42:32.188690 21487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:32.193099 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21405
W20260925 16:42:32.198381 21390 connection.cc:535] client connection to 127.0.0.1:35541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35541): Connection reset by peer (error 104)
2026-09-25T16:42:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:32 dist-test-slave-kxl8 krb5kdc[21504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:32 dist-test-slave-kxl8 krb5kdc[21504](info): set up 2 sockets
Sep 25 16:42:32 dist-test-slave-kxl8 krb5kdc[21504](info): commencing operation
krb5kdc: starting...
W20260925 16:42:33.217506 391 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:33 dist-test-slave-kxl8 krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:33.336410 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:33.444521 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:33.444819 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:33.444904 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:33.448820 21517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:33.448931 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:33.448976 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:33.449012 21517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:33.449046 21517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:33.454241 21517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:33.455284 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:33.456322 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:33.464591 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:33.464572 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:33.464645 21517 server_base.cc:1034] running on GCE node
W20260925 16:42:33.464572 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:33.465353 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:33.466394 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:33.467571 21517 hybrid_clock.cc:648] HybridClock initialized: now 1790354553467554 us; error 31 us; skew 500 ppm
Sep 25 16:42:33 dist-test-slave-kxl8 krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:33.470228 21517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:33.471061 21517 webserver.cc:460] Webserver started at http://127.0.0.1:45155/ using document root <none> and password file <none>
I20260925 16:42:33.471608 21517 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:33.471709 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:33.471932 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:33.474014 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d4054da0795f4ae2a47aff8490c4002e"
format_stamp: "Formatted at 2026-09-25 16:42:33 on dist-test-slave-kxl8"
server_key: "f89e5afbfbe42faef327ec8885767014"
server_key_iv: "d252a4cd4d1128a42203fb1ebdf9b4c5"
server_key_version: "encryptionkey@0"
I20260925 16:42:33.474527 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d4054da0795f4ae2a47aff8490c4002e"
format_stamp: "Formatted at 2026-09-25 16:42:33 on dist-test-slave-kxl8"
server_key: "f89e5afbfbe42faef327ec8885767014"
server_key_iv: "d252a4cd4d1128a42203fb1ebdf9b4c5"
server_key_version: "encryptionkey@0"
I20260925 16:42:33.478124 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:33.480522 21532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:33.481818 21517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:33.481967 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d4054da0795f4ae2a47aff8490c4002e"
format_stamp: "Formatted at 2026-09-25 16:42:33 on dist-test-slave-kxl8"
server_key: "f89e5afbfbe42faef327ec8885767014"
server_key_iv: "d252a4cd4d1128a42203fb1ebdf9b4c5"
server_key_version: "encryptionkey@0"
I20260925 16:42:33.482085 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:33.486517 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:33.489391 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:33.489584 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:33.496343 21517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40795
I20260925 16:42:33.496426 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40795 every 8 connection(s)
I20260925 16:42:33.497414 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:33.500296 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:33.501791 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21517
I20260925 16:42:33.501946 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:33.502249 391 external_mini_cluster.cc:1428] Setting key d2b470d1d1ce0584d90dc6a2af5c5a3e
I20260925 16:42:33.507027 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e: Bootstrap starting.
I20260925 16:42:33.507834 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:33.503718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46475 (local address 127.0.0.1:40795)
0925 16:42:33.504152 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:33.504173 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:33.504218 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:42:33.505056 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:33.505080 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:33.505488 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:33.505860 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:33.506620 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:33.506626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:33.506640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:33.506651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:33.506713 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:33.506739 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:33.506878 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:33.506984 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 16:42:33.507142 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260925 16:42:33.509375 21585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:33.510221 21585 log.cc:824] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:33.510593 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:33.512297 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e: No bootstrap required, opened a new log
I20260925 16:42:33.514463 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:33.512772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:53393 (local address 127.0.0.1:40795)
0925 16:42:33.512913 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:33.512917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:33.513118 (+ 201us) server_negotiation.cc:407] Connection header received
0925 16:42:33.513230 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:33.513232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:33.513260 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:33.513379 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:33.514050 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:33.514052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:33.514054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:33.514064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:33.514088 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:33.514098 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:33.514166 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:33.514271 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 16:42:33.514311 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 16:42:33.516518 21585 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260925 16:42:33.516762 21585 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:33.516819 21585 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4054da0795f4ae2a47aff8490c4002e, State: Initialized, Role: FOLLOWER
I20260925 16:42:33.517213 21585 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260925 16:42:33.517391 21585 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:33.517484 21585 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:33.517583 21585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:33.518500 21585 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260925 16:42:33.518837 21585 leader_election.cc:302] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4054da0795f4ae2a47aff8490c4002e; no voters:
I20260925 16:42:33.519188 21585 leader_election.cc:288] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:33.519445 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:33.519654 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [term 1 LEADER]: Becoming Leader. State: Replica: d4054da0795f4ae2a47aff8490c4002e, State: Running, Role: LEADER
I20260925 16:42:33.519935 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260925 16:42:33.520430 21585 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:33.521688 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4054da0795f4ae2a47aff8490c4002e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260925 16:42:33.521724 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4054da0795f4ae2a47aff8490c4002e. Latest consensus state: current_term: 1 leader_uuid: "d4054da0795f4ae2a47aff8490c4002e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4054da0795f4ae2a47aff8490c4002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260925 16:42:33.521817 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:33.521803 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:33.522274 21604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:33.524973 21604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:33.530300 21604 catalog_manager.cc:1347] Generated new cluster ID: dcb124ca0e48443690906e2c393b9acd
I20260925 16:42:33.530362 21604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:33.543906 21604 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:33.544250 21604 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:33.560940 21604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4054da0795f4ae2a47aff8490c4002e: Generated new TSK 0
I20260925 16:42:33.561873 21604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:33.567291 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21517
2026-09-25T16:42:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:33 dist-test-slave-kxl8 krb5kdc[21616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:33 dist-test-slave-kxl8 krb5kdc[21616](info): set up 2 sockets
Sep 25 16:42:33 dist-test-slave-kxl8 krb5kdc[21616](info): commencing operation
krb5kdc: starting...
W20260925 16:42:34.605613 391 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:34 dist-test-slave-kxl8 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:34.724534 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:34.831084 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:34.831382 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:34.831449 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:34.835398 21629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:34.835486 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:34.835525 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:34.835561 21629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:34.835596 21629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:34.840677 21629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:34.841677 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:34.842653 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:34.850952 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:34.850965 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:34.850980 21637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:34.851222 21629 server_base.cc:1034] running on GCE node
I20260925 16:42:34.851832 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:34.852869 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:34.854045 21629 hybrid_clock.cc:648] HybridClock initialized: now 1790354554854032 us; error 23 us; skew 500 ppm
Sep 25 16:42:34 dist-test-slave-kxl8 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:34.856669 21629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:34.857537 21629 webserver.cc:460] Webserver started at http://127.0.0.1:41689/ using document root <none> and password file <none>
I20260925 16:42:34.858026 21629 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:34.858080 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:34.858229 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:34.860117 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "8f897286b922401681ef53a94197e1e6"
format_stamp: "Formatted at 2026-09-25 16:42:34 on dist-test-slave-kxl8"
server_key: "823662ee2991dcfe38d32d0931dee1bb"
server_key_iv: "84a29a6671ed9114a4c8c5786dfa1142"
server_key_version: "encryptionkey@0"
I20260925 16:42:34.860634 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "8f897286b922401681ef53a94197e1e6"
format_stamp: "Formatted at 2026-09-25 16:42:34 on dist-test-slave-kxl8"
server_key: "823662ee2991dcfe38d32d0931dee1bb"
server_key_iv: "84a29a6671ed9114a4c8c5786dfa1142"
server_key_version: "encryptionkey@0"
I20260925 16:42:34.864037 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 16:42:34.866241 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:34.867411 21629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:34.867550 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "8f897286b922401681ef53a94197e1e6"
format_stamp: "Formatted at 2026-09-25 16:42:34 on dist-test-slave-kxl8"
server_key: "823662ee2991dcfe38d32d0931dee1bb"
server_key_iv: "84a29a6671ed9114a4c8c5786dfa1142"
server_key_version: "encryptionkey@0"
I20260925 16:42:34.867676 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:34.889789 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:34.892475 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:34.892668 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:34.899305 21629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41771
I20260925 16:42:34.899318 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41771 every 8 connection(s)
I20260925 16:42:34.900209 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:34.902948 21697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:34.908155 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6: Bootstrap starting.
I20260925 16:42:34.910241 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21629
I20260925 16:42:34.910377 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:34.910689 21697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:34.910710 391 external_mini_cluster.cc:1428] Setting key a81c48c403bbf6d412f907231bf4cb91
I20260925 16:42:34.911552 21697 log.cc:824] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:34.913419 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6: No bootstrap required, opened a new log
I20260925 16:42:34.916644 21697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260925 16:42:34.916878 21697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:34.916994 21697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f897286b922401681ef53a94197e1e6, State: Initialized, Role: FOLLOWER
I20260925 16:42:34.917500 21697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260925 16:42:34.917660 21697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:34.917747 21697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:34.917863 21697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:34.918833 21697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
Sep 25 16:42:34 dist-test-slave-kxl8 krb5kdc[21616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:42:34.919203 21697 leader_election.cc:302] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f897286b922401681ef53a94197e1e6; no voters:
I20260925 16:42:34.919539 21697 leader_election.cc:288] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:34.919642 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:34.919884 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8f897286b922401681ef53a94197e1e6, State: Running, Role: LEADER
I20260925 16:42:34.920261 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260925 16:42:34.920770 21697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:34.922111 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f897286b922401681ef53a94197e1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } } }
I20260925 16:42:34.922241 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:34.922585 21713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:34.923115 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f897286b922401681ef53a94197e1e6. Latest consensus state: current_term: 1 leader_uuid: "8f897286b922401681ef53a94197e1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f897286b922401681ef53a94197e1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } } }
I20260925 16:42:34.923856 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:34.925648 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:34.911823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49831 (local address 127.0.0.1:41771)
0925 16:42:34.912272 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:34.912298 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:34.912331 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:42:34.912916 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:34.912932 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:34.913252 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:34.913635 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:34.914395 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:34.915331 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:34.916159 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:34.916615 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:34.916790 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:34.919798 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:34.919805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:34.919822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:34.919860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:34.921960 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:34.922974 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:34.922981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:34.922986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:34.923133 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:34.923539 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:34.923542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:34.923544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:34.924036 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:34.924241 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:34.924650 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 16:42:34.924890 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260925 16:42:34.925748 21713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:34.928570 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:34.932255 21713 catalog_manager.cc:1347] Generated new cluster ID: d9c2ba33e5b3459a9de8ecc2e0a81d9d
I20260925 16:42:34.932314 21713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:34.938566 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:34.930727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:33065 (local address 127.0.0.1:41771)
0925 16:42:34.930835 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:34.930839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:34.931062 (+ 223us) server_negotiation.cc:407] Connection header received
0925 16:42:34.931184 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:34.931187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:34.931231 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:34.931336 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:34.932138 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:34.932744 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:34.933630 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:34.933865 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:34.933974 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:34.935335 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:34.935338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:34.935341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:34.935371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:34.936807 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:34.937309 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:34.937313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:34.937314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:34.937397 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:34.937737 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:34.937740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:34.937741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:34.937903 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:34.938057 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:34.938326 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 16:42:34.938414 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 16:42:34.949319 21713 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:34.950258 21713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:34.956436 21713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f897286b922401681ef53a94197e1e6: Generated new TSK 0
I20260925 16:42:34.957233 21713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:34.965826 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21629
W20260925 16:42:34.971172 21614 connection.cc:535] client connection to 127.0.0.1:41771 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41771): Connection reset by peer (error 104)
2026-09-25T16:42:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:35 dist-test-slave-kxl8 krb5kdc[21728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:35 dist-test-slave-kxl8 krb5kdc[21728](info): set up 2 sockets
Sep 25 16:42:35 dist-test-slave-kxl8 krb5kdc[21728](info): commencing operation
krb5kdc: starting...
W20260925 16:42:36.009505 391 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:36 dist-test-slave-kxl8 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:36.128355 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:36.235168 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:36.235420 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:36.235468 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:36.239288 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:36.239341 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:36.239359 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:36.239377 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:36.239391 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:36.244372 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:36.245349 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:36.246308 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:36.254851 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:36.254895 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:36.254904 21741 server_base.cc:1034] running on GCE node
W20260925 16:42:36.254851 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:36.255744 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:36.256881 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:36.258039 21741 hybrid_clock.cc:648] HybridClock initialized: now 1790354556258028 us; error 27 us; skew 500 ppm
Sep 25 16:42:36 dist-test-slave-kxl8 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:36.260706 21741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:36.261649 21741 webserver.cc:460] Webserver started at http://127.0.0.1:39917/ using document root <none> and password file <none>
I20260925 16:42:36.262204 21741 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:36.262295 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:36.262528 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:36.264564 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "4cf2c014a6d447f6936e3808304b9911"
format_stamp: "Formatted at 2026-09-25 16:42:36 on dist-test-slave-kxl8"
server_key: "1da8435f9e1e5c8b1a05f13fde21556a"
server_key_iv: "0e66bdb4d4edd29beeb4b26de3cb20dc"
server_key_version: "encryptionkey@0"
I20260925 16:42:36.265118 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "4cf2c014a6d447f6936e3808304b9911"
format_stamp: "Formatted at 2026-09-25 16:42:36 on dist-test-slave-kxl8"
server_key: "1da8435f9e1e5c8b1a05f13fde21556a"
server_key_iv: "0e66bdb4d4edd29beeb4b26de3cb20dc"
server_key_version: "encryptionkey@0"
I20260925 16:42:36.268750 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:42:36.271107 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:36.272375 21741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:36.272537 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "4cf2c014a6d447f6936e3808304b9911"
format_stamp: "Formatted at 2026-09-25 16:42:36 on dist-test-slave-kxl8"
server_key: "1da8435f9e1e5c8b1a05f13fde21556a"
server_key_iv: "0e66bdb4d4edd29beeb4b26de3cb20dc"
server_key_version: "encryptionkey@0"
I20260925 16:42:36.272650 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:36.286561 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:36.289341 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:36.289546 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:36.296492 21741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38733
I20260925 16:42:36.296511 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38733 every 8 connection(s)
I20260925 16:42:36.297467 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:36.300077 21809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:36.304029 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21741
I20260925 16:42:36.304171 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:36.304430 391 external_mini_cluster.cc:1428] Setting key 37826975b43476a1302fdb15f40b7f40
I20260925 16:42:36.305686 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911: Bootstrap starting.
I20260925 16:42:36.307826 21809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:36.308537 21809 log.cc:824] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:36.310295 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911: No bootstrap required, opened a new log
I20260925 16:42:36.312920 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:36.305747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:33957 (local address 127.0.0.1:38733)
0925 16:42:36.306150 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:36.306164 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:36.306199 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:42:36.306813 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:36.306828 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:36.307153 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:36.307484 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:36.307873 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:36.308938 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:36.309858 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:36.310308 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:36.310480 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:36.311338 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:36.311343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:36.311355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:36.311715 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:36.311763 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:36.311784 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:36.311919 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:36.312125 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 16:42:36.312257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260925 16:42:36.314059 21809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 16:42:36.314306 21809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:36.314352 21809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf2c014a6d447f6936e3808304b9911, State: Initialized, Role: FOLLOWER
I20260925 16:42:36.314791 21809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 16:42:36.314968 21809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:36.315032 21809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:36.315200 21809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:36.315864 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:36.316377 21809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 16:42:36.316773 21809 leader_election.cc:302] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf2c014a6d447f6936e3808304b9911; no voters:
I20260925 16:42:36.317130 21809 leader_election.cc:288] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:36.317194 21815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:36.317461 21815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [term 1 LEADER]: Becoming Leader. State: Replica: 4cf2c014a6d447f6936e3808304b9911, State: Running, Role: LEADER
I20260925 16:42:36.317800 21815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 16:42:36.318418 21809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:36.318991 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf2c014a6d447f6936e3808304b9911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } } }
I20260925 16:42:36.319175 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:36.319386 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf2c014a6d447f6936e3808304b9911. Latest consensus state: current_term: 1 leader_uuid: "4cf2c014a6d447f6936e3808304b9911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf2c014a6d447f6936e3808304b9911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } } }
I20260925 16:42:36.319566 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:36.319597 21827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:36.322724 21827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:36.324842 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:36.318031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:50387 (local address 127.0.0.1:38733)
0925 16:42:36.318135 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:36.318139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:36.320642 (+ 2503us) server_negotiation.cc:407] Connection header received
0925 16:42:36.320927 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:36.320931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:36.320990 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:36.321065 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:36.321340 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:36.322128 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:36.322938 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:36.323286 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:36.323389 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:36.324387 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:36.324394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:36.324398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:36.324414 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:36.324442 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:36.324455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:36.324538 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:36.324662 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 16:42:36.324694 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:42:36.328390 21827 catalog_manager.cc:1347] Generated new cluster ID: ab8cdf0f47e34738ae7fbcecc88e15ab
I20260925 16:42:36.328459 21827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:36.350095 21827 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:36.350997 21827 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:36.356304 21827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cf2c014a6d447f6936e3808304b9911: Generated new TSK 0
I20260925 16:42:36.357060 21827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:36.362253 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21741
W20260925 16:42:36.367436 21726 connection.cc:535] client connection to 127.0.0.1:38733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38733): Connection reset by peer (error 104)
2026-09-25T16:42:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:36 dist-test-slave-kxl8 krb5kdc[21840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:36 dist-test-slave-kxl8 krb5kdc[21840](info): set up 2 sockets
Sep 25 16:42:36 dist-test-slave-kxl8 krb5kdc[21840](info): commencing operation
krb5kdc: starting...
W20260925 16:42:37.397430 391 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:37 dist-test-slave-kxl8 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:42:37.520644 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:37.631794 21853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:37.632088 21853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:37.632161 21853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:37.636096 21853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:37.636176 21853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:37.636214 21853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:37.636252 21853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:37.636284 21853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:37.641388 21853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:37.642396 21853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:37.643405 21853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:37.651732 21859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:37.651754 21858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:37.651772 21853 server_base.cc:1034] running on GCE node
W20260925 16:42:37.651908 21861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:37.652617 21853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:37.653740 21853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:37.654944 21853 hybrid_clock.cc:648] HybridClock initialized: now 1790354557654926 us; error 28 us; skew 500 ppm
Sep 25 16:42:37 dist-test-slave-kxl8 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:37.657763 21853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:42:37.658653 21853 webserver.cc:460] Webserver started at http://127.0.0.1:45189/ using document root <none> and password file <none>
I20260925 16:42:37.659200 21853 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:37.659287 21853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:37.659511 21853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:37.661597 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d54eb5e11ef145efb684caaadedeb439"
format_stamp: "Formatted at 2026-09-25 16:42:37 on dist-test-slave-kxl8"
server_key: "d7cee8dc5576a83226f4c7727a111d0f"
server_key_iv: "3ffd3f3e85bb4e0ccd66d9c5480c14a7"
server_key_version: "encryptionkey@0"
I20260925 16:42:37.662142 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d54eb5e11ef145efb684caaadedeb439"
format_stamp: "Formatted at 2026-09-25 16:42:37 on dist-test-slave-kxl8"
server_key: "d7cee8dc5576a83226f4c7727a111d0f"
server_key_iv: "3ffd3f3e85bb4e0ccd66d9c5480c14a7"
server_key_version: "encryptionkey@0"
I20260925 16:42:37.665746 21853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:42:37.668027 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:37.669173 21853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:37.669332 21853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d54eb5e11ef145efb684caaadedeb439"
format_stamp: "Formatted at 2026-09-25 16:42:37 on dist-test-slave-kxl8"
server_key: "d7cee8dc5576a83226f4c7727a111d0f"
server_key_iv: "3ffd3f3e85bb4e0ccd66d9c5480c14a7"
server_key_version: "encryptionkey@0"
I20260925 16:42:37.669445 21853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:37.673949 21853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:37.677098 21853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:37.677336 21853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:37.683768 21853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38897
I20260925 16:42:37.683816 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38897 every 8 connection(s)
I20260925 16:42:37.684671 21853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:37.687192 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21853
I20260925 16:42:37.687317 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:37.687525 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:37.687675 391 external_mini_cluster.cc:1428] Setting key fde4c2f67f5c82180cdeed58503b3725
I20260925 16:42:37.693192 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:37.689023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:35551 (local address 127.0.0.1:38897)
0925 16:42:37.689579 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:37.689597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:37.689727 (+ 130us) server_negotiation.cc:407] Connection header received
0925 16:42:37.690687 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:37.690704 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:37.690979 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:37.691242 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:37.692030 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:37.692035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:37.692045 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:37.692057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:37.692110 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:37.692129 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:37.692251 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:37.692353 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 16:42:37.692538 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":159,"threads_started":1}
I20260925 16:42:37.694052 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439: Bootstrap starting.
I20260925 16:42:37.695782 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:37.696328 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:37.697211 21921 log.cc:824] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:37.699600 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439: No bootstrap required, opened a new log
I20260925 16:42:37.699752 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:37.698060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46207 (local address 127.0.0.1:38897)
0925 16:42:37.698180 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:37.698184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:37.698470 (+ 286us) server_negotiation.cc:407] Connection header received
0925 16:42:37.698581 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:37.698583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:37.698611 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:37.698667 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:37.699304 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:37.699306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:37.699308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:37.699318 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:37.699341 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:37.699351 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:37.699416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:37.699528 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 16:42:37.699560 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:37.703056 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260925 16:42:37.703292 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:37.703362 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54eb5e11ef145efb684caaadedeb439, State: Initialized, Role: FOLLOWER
I20260925 16:42:37.703802 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260925 16:42:37.703948 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:37.704016 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:37.704111 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:37.704986 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260925 16:42:37.705358 21921 leader_election.cc:302] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d54eb5e11ef145efb684caaadedeb439; no voters:
I20260925 16:42:37.705663 21921 leader_election.cc:288] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:37.705817 21931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:37.706028 21931 raft_consensus.cc:686] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [term 1 LEADER]: Becoming Leader. State: Replica: d54eb5e11ef145efb684caaadedeb439, State: Running, Role: LEADER
I20260925 16:42:37.706560 21931 consensus_queue.cc:237] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260925 16:42:37.706882 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:37.708274 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d54eb5e11ef145efb684caaadedeb439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260925 16:42:37.708290 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d54eb5e11ef145efb684caaadedeb439. Latest consensus state: current_term: 1 leader_uuid: "d54eb5e11ef145efb684caaadedeb439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54eb5e11ef145efb684caaadedeb439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260925 16:42:37.708482 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:37.708736 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:37.708998 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:37.711980 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:37.716733 21943 catalog_manager.cc:1347] Generated new cluster ID: f00b931e2fef4b0a9e62736cf0846c7a
I20260925 16:42:37.716794 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:37.729250 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:37.729585 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:37.734156 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d54eb5e11ef145efb684caaadedeb439: Generated new TSK 0
I20260925 16:42:37.734870 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:37.746908 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21853
2026-09-25T16:42:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:37 dist-test-slave-kxl8 krb5kdc[21952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:37 dist-test-slave-kxl8 krb5kdc[21952](info): set up 2 sockets
Sep 25 16:42:37 dist-test-slave-kxl8 krb5kdc[21952](info): commencing operation
krb5kdc: starting...
W20260925 16:42:38.829404 391 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:38 dist-test-slave-kxl8 krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:38.949685 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44311
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:39.060104 21965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:39.060408 21965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:39.060509 21965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:39.064589 21965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:39.064679 21965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:39.064718 21965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:39.064757 21965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:39.064785 21965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:39.070050 21965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:39.071036 21965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:39.072042 21965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:39.080283 21970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:39.080488 21973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:39.080307 21971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:39.080734 21965 server_base.cc:1034] running on GCE node
I20260925 16:42:39.081338 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:39.082446 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:39.083634 21965 hybrid_clock.cc:648] HybridClock initialized: now 1790354559083608 us; error 49 us; skew 500 ppm
Sep 25 16:42:39 dist-test-slave-kxl8 krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:39.086295 21965 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:39.087141 21965 webserver.cc:460] Webserver started at http://127.0.97.254:38171/ using document root <none> and password file <none>
I20260925 16:42:39.087687 21965 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:39.087790 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:39.087972 21965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:39.090005 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "c96164bdd1b2473792d31befe4db69b6"
format_stamp: "Formatted at 2026-09-25 16:42:39 on dist-test-slave-kxl8"
server_key: "0e59096e999a6cfd1acf287cfac090ef"
server_key_iv: "238154a23993e481eb9a865900ea5e4b"
server_key_version: "encryptionkey@0"
I20260925 16:42:39.090497 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "c96164bdd1b2473792d31befe4db69b6"
format_stamp: "Formatted at 2026-09-25 16:42:39 on dist-test-slave-kxl8"
server_key: "0e59096e999a6cfd1acf287cfac090ef"
server_key_iv: "238154a23993e481eb9a865900ea5e4b"
server_key_version: "encryptionkey@0"
I20260925 16:42:39.094144 21965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:39.096344 21980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:39.097564 21965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:39.097703 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "c96164bdd1b2473792d31befe4db69b6"
format_stamp: "Formatted at 2026-09-25 16:42:39 on dist-test-slave-kxl8"
server_key: "0e59096e999a6cfd1acf287cfac090ef"
server_key_iv: "238154a23993e481eb9a865900ea5e4b"
server_key_version: "encryptionkey@0"
I20260925 16:42:39.097815 21965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:39.138031 21965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:39.141166 21965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:39.141407 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:39.148730 21965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44311
I20260925 16:42:39.148758 22032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44311 every 8 connection(s)
I20260925 16:42:39.149699 21965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:39.152341 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:39.156411 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 21965
I20260925 16:42:39.156559 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:39.156838 391 external_mini_cluster.cc:1428] Setting key 24732344b3b046d730e50256d0eabac5
I20260925 16:42:39.158265 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6: Bootstrap starting.
I20260925 16:42:39.160447 22033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:39.161170 22033 log.cc:824] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:39.163085 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6: No bootstrap required, opened a new log
Sep 25 16:42:39 dist-test-slave-kxl8 krb5kdc[21952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:39.167503 22033 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } }
I20260925 16:42:39.167815 22033 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:39.167886 22033 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96164bdd1b2473792d31befe4db69b6, State: Initialized, Role: FOLLOWER
I20260925 16:42:39.168398 22033 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } }
I20260925 16:42:39.168527 22033 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:39.168609 22033 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:39.168727 22033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:39.169723 22033 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } }
I20260925 16:42:39.170073 22033 leader_election.cc:302] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96164bdd1b2473792d31befe4db69b6; no voters:
I20260925 16:42:39.170380 22033 leader_election.cc:288] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:39.170476 22038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:39.170699 22038 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [term 1 LEADER]: Becoming Leader. State: Replica: c96164bdd1b2473792d31befe4db69b6, State: Running, Role: LEADER
I20260925 16:42:39.171087 22038 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } }
I20260925 16:42:39.171810 22033 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:39.172837 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96164bdd1b2473792d31befe4db69b6. Latest consensus state: current_term: 1 leader_uuid: "c96164bdd1b2473792d31befe4db69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } } }
I20260925 16:42:39.172976 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:39.173061 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:39.158308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44799 (local address 127.0.97.254:44311)
0925 16:42:39.158737 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:39.158752 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:39.158786 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:42:39.159463 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:39.159482 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:39.159820 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:39.160267 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:39.161073 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:39.162409 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:39.163287 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:39.163959 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:39.164124 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:39.167138 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:39.167144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:39.167162 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:39.167197 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:39.169255 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:39.169830 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:39.169837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:39.169846 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:39.169987 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:39.170344 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:39.170347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:39.170350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:39.170942 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:39.171157 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:39.171801 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 16:42:39.172027 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260925 16:42:39.173533 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96164bdd1b2473792d31befe4db69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96164bdd1b2473792d31befe4db69b6" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44311 } } }
I20260925 16:42:39.173683 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:39.174175 22047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:39.176328 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:39.177160 22047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:39.183130 22047 catalog_manager.cc:1347] Generated new cluster ID: 60bd62dbac944527ba8e1bd30b9b2e49
I20260925 16:42:39.183187 22047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:39.189870 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:39.179160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36455 (local address 127.0.97.254:44311)
0925 16:42:39.179278 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:39.179282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:39.179495 (+ 213us) server_negotiation.cc:407] Connection header received
0925 16:42:39.179667 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:39.179669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:39.179711 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:39.179799 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:39.180422 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:39.181061 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:39.182111 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:39.182434 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:39.182522 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:39.186409 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:39.186413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:39.186415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:39.186440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:39.188080 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:39.188615 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:39.188619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:39.188620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:39.188696 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:39.189017 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:39.189020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:39.189021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:39.189206 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:39.189378 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:39.189636 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 16:42:39.189723 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:42:39.205521 22047 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:39.206430 22047 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:39.216043 22047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96164bdd1b2473792d31befe4db69b6: Generated new TSK 0
I20260925 16:42:39.216827 22047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:39.226518 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 21965
W20260925 16:42:39.232863 21950 connection.cc:535] client connection to 127.0.97.254:44311 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.254:44311): Connection reset by peer (error 104)
2026-09-25T16:42:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:39 dist-test-slave-kxl8 krb5kdc[22064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:39 dist-test-slave-kxl8 krb5kdc[22064](info): set up 2 sockets
Sep 25 16:42:39 dist-test-slave-kxl8 krb5kdc[22064](info): commencing operation
krb5kdc: starting...
W20260925 16:42:40.277462 391 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:40 dist-test-slave-kxl8 krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:40.397681 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42261
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:40.503909 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:40.504192 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:40.504285 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:40.508592 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:40.508668 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:40.508703 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:40.508731 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:40.508767 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:40.514602 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:40.515667 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:40.516673 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:40.525298 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:40.525231 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:40.525228 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:40.525254 22077 server_base.cc:1034] running on GCE node
I20260925 16:42:40.526046 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:40.527175 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:40.528363 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790354560528354 us; error 27 us; skew 500 ppm
Sep 25 16:42:40 dist-test-slave-kxl8 krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:40.531190 22077 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:40.532291 22077 webserver.cc:460] Webserver started at http://127.0.97.254:42919/ using document root <none> and password file <none>
I20260925 16:42:40.532900 22077 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:40.532999 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:40.533249 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:40.535507 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "c7215e0a944b46fb9d10d62a4594cad4"
format_stamp: "Formatted at 2026-09-25 16:42:40 on dist-test-slave-kxl8"
server_key: "e1dbaad0985421449af6863f877269f0"
server_key_iv: "b791990645275406758f5c68e4b10b78"
server_key_version: "encryptionkey@0"
I20260925 16:42:40.536125 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "c7215e0a944b46fb9d10d62a4594cad4"
format_stamp: "Formatted at 2026-09-25 16:42:40 on dist-test-slave-kxl8"
server_key: "e1dbaad0985421449af6863f877269f0"
server_key_iv: "b791990645275406758f5c68e4b10b78"
server_key_version: "encryptionkey@0"
I20260925 16:42:40.540141 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:42:40.542716 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:40.543932 22077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:40.544075 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "c7215e0a944b46fb9d10d62a4594cad4"
format_stamp: "Formatted at 2026-09-25 16:42:40 on dist-test-slave-kxl8"
server_key: "e1dbaad0985421449af6863f877269f0"
server_key_iv: "b791990645275406758f5c68e4b10b78"
server_key_version: "encryptionkey@0"
I20260925 16:42:40.544188 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:40.555970 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:40.558694 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:40.558887 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:40.566231 22077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42261
I20260925 16:42:40.566282 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42261 every 8 connection(s)
I20260925 16:42:40.567170 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:40.570178 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:40.573493 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22077
I20260925 16:42:40.573650 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:40.573918 391 external_mini_cluster.cc:1428] Setting key cbf180fab27e0b6eb0dcac15ad5843da
I20260925 16:42:40.576556 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4: Bootstrap starting.
I20260925 16:42:40.578944 22145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:40.579728 22145 log.cc:824] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:40.581753 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4: No bootstrap required, opened a new log
I20260925 16:42:40.582686 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:40.575310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33359 (local address 127.0.97.254:42261)
0925 16:42:40.575781 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:40.575796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:40.575832 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:40.576428 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:40.576444 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:40.576821 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:40.577154 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:40.577602 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:40.578551 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:40.579491 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:40.580010 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:40.580168 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:40.581075 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:40.581080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:40.581092 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:40.581527 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:40.581580 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:40.581603 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:40.581739 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:40.581900 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 16:42:40.582039 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":149,"threads_started":1}
I20260925 16:42:40.585856 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:40.586339 22145 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } }
I20260925 16:42:40.586565 22145 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:40.586639 22145 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7215e0a944b46fb9d10d62a4594cad4, State: Initialized, Role: FOLLOWER
I20260925 16:42:40.587164 22145 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } }
I20260925 16:42:40.587349 22145 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:40.587430 22145 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:40.587556 22145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:40.588884 22145 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } }
I20260925 16:42:40.589268 22145 leader_election.cc:302] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7215e0a944b46fb9d10d62a4594cad4; no voters:
I20260925 16:42:40.589686 22145 leader_election.cc:288] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:40.589785 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:40.590005 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [term 1 LEADER]: Becoming Leader. State: Replica: c7215e0a944b46fb9d10d62a4594cad4, State: Running, Role: LEADER
I20260925 16:42:40.590392 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } }
I20260925 16:42:40.591086 22145 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:40.592378 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7215e0a944b46fb9d10d62a4594cad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } } }
I20260925 16:42:40.592384 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7215e0a944b46fb9d10d62a4594cad4. Latest consensus state: current_term: 1 leader_uuid: "c7215e0a944b46fb9d10d62a4594cad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7215e0a944b46fb9d10d62a4594cad4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42261 } } }
I20260925 16:42:40.592612 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:40.592617 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:40.592945 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:40.587998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45333 (local address 127.0.97.254:42261)
0925 16:42:40.588139 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:40.588144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:40.588409 (+ 265us) server_negotiation.cc:407] Connection header received
0925 16:42:40.588550 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:40.588552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:40.588589 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:40.588652 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:40.589039 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:40.589650 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:40.590746 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:40.591021 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:40.591112 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:40.592205 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:40.592209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:40.592212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:40.592226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:40.592257 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:40.592270 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:40.592338 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:40.592593 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 16:42:40.592634 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:42:40.593119 22165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:40.596324 22165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:40.605953 22165 catalog_manager.cc:1347] Generated new cluster ID: bef756f59ce54346b7f448dc15cac66e
I20260925 16:42:40.606043 22165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:40.618460 22165 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:40.619490 22165 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:40.632706 22165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7215e0a944b46fb9d10d62a4594cad4: Generated new TSK 0
I20260925 16:42:40.633565 22165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:40.643402 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22077
W20260925 16:42:40.648799 22062 connection.cc:535] client connection to 127.0.97.254:42261 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.254:42261): Connection reset by peer (error 104)
2026-09-25T16:42:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:40 dist-test-slave-kxl8 krb5kdc[22176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:40 dist-test-slave-kxl8 krb5kdc[22176](info): set up 2 sockets
Sep 25 16:42:40 dist-test-slave-kxl8 krb5kdc[22176](info): commencing operation
krb5kdc: starting...
W20260925 16:42:41.697603 391 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:41 dist-test-slave-kxl8 krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:41.815786 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44689
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:42677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:41.923121 22189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:41.923411 22189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:41.923489 22189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:41.927429 22189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:41.927521 22189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:41.927562 22189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:41.927620 22189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:41.927656 22189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:41.932809 22189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:42677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:44689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:41.933885 22189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:41.934895 22189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:41.943184 22194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:41.943195 22195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:41.943184 22197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:41.943589 22189 server_base.cc:1034] running on GCE node
I20260925 16:42:41.944105 22189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:41.945191 22189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:41.946350 22189 hybrid_clock.cc:648] HybridClock initialized: now 1790354561946330 us; error 32 us; skew 500 ppm
Sep 25 16:42:41 dist-test-slave-kxl8 krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:41.949100 22189 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:41.949986 22189 webserver.cc:460] Webserver started at http://127.0.97.254:36189/ using document root <none> and password file <none>
I20260925 16:42:41.950485 22189 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:41.950543 22189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:41.950773 22189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:41.952744 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "d126e292dce046daa1423aab9d03fee4"
format_stamp: "Formatted at 2026-09-25 16:42:41 on dist-test-slave-kxl8"
server_key: "889634049a38da9ac251dbd58972ec53"
server_key_iv: "51d5d18aaccf17ad3ec55b8fdb735b46"
server_key_version: "encryptionkey@0"
I20260925 16:42:41.953249 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "d126e292dce046daa1423aab9d03fee4"
format_stamp: "Formatted at 2026-09-25 16:42:41 on dist-test-slave-kxl8"
server_key: "889634049a38da9ac251dbd58972ec53"
server_key_iv: "51d5d18aaccf17ad3ec55b8fdb735b46"
server_key_version: "encryptionkey@0"
I20260925 16:42:41.956779 22189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:42:41.959133 22204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:41.960320 22189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:41.960564 22189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "d126e292dce046daa1423aab9d03fee4"
format_stamp: "Formatted at 2026-09-25 16:42:41 on dist-test-slave-kxl8"
server_key: "889634049a38da9ac251dbd58972ec53"
server_key_iv: "51d5d18aaccf17ad3ec55b8fdb735b46"
server_key_version: "encryptionkey@0"
I20260925 16:42:41.960677 22189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:41.965106 22189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:41.968288 22189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:41.968484 22189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:41.975379 22189 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:44689
I20260925 16:42:41.975395 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:44689 every 8 connection(s)
I20260925 16:42:41.976311 22189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:41.979027 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:41.981259 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22189
I20260925 16:42:41.981427 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:41.981724 391 external_mini_cluster.cc:1428] Setting key a2bc1e2eb012f0b0e87bf1ffa358c679
I20260925 16:42:41.985086 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4: Bootstrap starting.
I20260925 16:42:41.986801 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:41.983027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37355 (local address 127.0.97.254:44689)
0925 16:42:41.983425 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:41.983440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:41.983481 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:42:41.984150 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:41.984166 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:41.984442 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:41.984728 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:41.985553 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:41.985557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:41.985569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:41.985578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:41.985629 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:41.985647 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:41.985752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:41.985854 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 16:42:41.986017 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260925 16:42:41.987639 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:41.989141 22257 log.cc:824] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:41.989502 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:41.991034 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4: No bootstrap required, opened a new log
I20260925 16:42:41.992945 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:41.991458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43499 (local address 127.0.97.254:44689)
0925 16:42:41.991577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:41.991581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:41.991702 (+ 121us) server_negotiation.cc:407] Connection header received
0925 16:42:41.991814 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:41.991816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:41.991842 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:41.991903 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:41.992559 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:41.992561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:41.992563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:41.992572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:41.992595 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:41.992604 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:41.992669 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:41.992768 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 16:42:41.992797 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:42:41.995225 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } }
I20260925 16:42:41.995426 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:41.995500 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d126e292dce046daa1423aab9d03fee4, State: Initialized, Role: FOLLOWER
I20260925 16:42:41.995930 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } }
I20260925 16:42:41.996076 22257 raft_consensus.cc:388] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:41.996141 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:41.996232 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:41.997082 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } }
I20260925 16:42:41.997448 22257 leader_election.cc:302] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d126e292dce046daa1423aab9d03fee4; no voters:
I20260925 16:42:41.997731 22257 leader_election.cc:288] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:41.997848 22267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:41.998095 22267 raft_consensus.cc:686] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [term 1 LEADER]: Becoming Leader. State: Replica: d126e292dce046daa1423aab9d03fee4, State: Running, Role: LEADER
I20260925 16:42:41.998377 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } }
I20260925 16:42:41.998912 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:41.999920 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d126e292dce046daa1423aab9d03fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } } }
I20260925 16:42:42.000036 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:42.000303 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d126e292dce046daa1423aab9d03fee4. Latest consensus state: current_term: 1 leader_uuid: "d126e292dce046daa1423aab9d03fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d126e292dce046daa1423aab9d03fee4" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 44689 } } }
I20260925 16:42:42.000394 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:42.000412 22276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:42.003576 22276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:42.009109 22276 catalog_manager.cc:1347] Generated new cluster ID: 38f073523f0e4c3c903450ee3957df2d
I20260925 16:42:42.009172 22276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:42.018568 22276 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:42.018886 22276 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:42.026283 22276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d126e292dce046daa1423aab9d03fee4: Generated new TSK 0
I20260925 16:42:42.027022 22276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:42.039856 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22189
2026-09-25T16:42:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:42 dist-test-slave-kxl8 krb5kdc[22288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:42 dist-test-slave-kxl8 krb5kdc[22288](info): set up 2 sockets
Sep 25 16:42:42 dist-test-slave-kxl8 krb5kdc[22288](info): commencing operation
krb5kdc: starting...
W20260925 16:42:43.077649 391 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:43.200312 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:46827
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:43.307498 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:43.307801 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:43.307889 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:43.311889 22301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:43.311969 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:43.312023 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:43.312063 22301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:43.312117 22301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:43.317529 22301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:43.318570 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:43.319562 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:43.327764 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:43.327795 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:43.327818 22301 server_base.cc:1034] running on GCE node
W20260925 16:42:43.327772 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:43.328562 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:43.329704 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:43.330892 22301 hybrid_clock.cc:648] HybridClock initialized: now 1790354563330876 us; error 29 us; skew 500 ppm
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:43.339888 22301 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:43.340775 22301 webserver.cc:460] Webserver started at http://127.0.97.254:43269/ using document root <none> and password file <none>
I20260925 16:42:43.341403 22301 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:43.341492 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:43.341723 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:43.343729 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "dd6095675a9442d08651617b089590ea"
format_stamp: "Formatted at 2026-09-25 16:42:43 on dist-test-slave-kxl8"
server_key: "0ca787918d941547ac608387b8d30d33"
server_key_iv: "f96b0510bcecb4df16d6d1686af88c67"
server_key_version: "encryptionkey@0"
I20260925 16:42:43.344282 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "dd6095675a9442d08651617b089590ea"
format_stamp: "Formatted at 2026-09-25 16:42:43 on dist-test-slave-kxl8"
server_key: "0ca787918d941547ac608387b8d30d33"
server_key_iv: "f96b0510bcecb4df16d6d1686af88c67"
server_key_version: "encryptionkey@0"
I20260925 16:42:43.347867 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:42:43.350179 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:43.351358 22301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:43.351521 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "dd6095675a9442d08651617b089590ea"
format_stamp: "Formatted at 2026-09-25 16:42:43 on dist-test-slave-kxl8"
server_key: "0ca787918d941547ac608387b8d30d33"
server_key_iv: "f96b0510bcecb4df16d6d1686af88c67"
server_key_version: "encryptionkey@0"
I20260925 16:42:43.351632 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:43.364253 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:43.367477 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:43.367678 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:43.374344 22301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:46827
I20260925 16:42:43.374382 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:46827 every 8 connection(s)
I20260925 16:42:43.375248 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:43.376070 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22301
I20260925 16:42:43.376200 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:43.376519 391 external_mini_cluster.cc:1428] Setting key 268dadbba7be3f6d864aa9ad92f92719
I20260925 16:42:43.378258 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:43.383502 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea: Bootstrap starting.
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:43.385982 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:43.386919 22369 log.cc:824] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:43.389045 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea: No bootstrap required, opened a new log
I20260925 16:42:43.390618 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:43.377875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60915 (local address 127.0.97.254:46827)
0925 16:42:43.378447 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:43.378466 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:43.378532 (+ 66us) server_negotiation.cc:407] Connection header received
0925 16:42:43.379145 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:43.379160 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:43.379472 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:43.379825 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:43.380615 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:43.381680 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:43.382519 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:43.382974 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:43.383133 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:43.385811 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:43.385820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:43.385840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:43.385878 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:43.387688 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:43.388081 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:43.388086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:43.388092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:43.388219 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:43.388459 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:43.388462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:43.388463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:43.388933 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:43.389141 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:43.389440 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 16:42:43.389725 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":132,"threads_started":1}
I20260925 16:42:43.393543 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:43.393445 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } }
I20260925 16:42:43.393728 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:43.393790 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6095675a9442d08651617b089590ea, State: Initialized, Role: FOLLOWER
I20260925 16:42:43.394225 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } }
I20260925 16:42:43.394340 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:43.394416 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:43.394522 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:43.395452 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } }
I20260925 16:42:43.395823 22369 leader_election.cc:302] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6095675a9442d08651617b089590ea; no voters:
I20260925 16:42:43.396162 22369 leader_election.cc:288] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:43.396302 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:43.396552 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [term 1 LEADER]: Becoming Leader. State: Replica: dd6095675a9442d08651617b089590ea, State: Running, Role: LEADER
I20260925 16:42:43.396842 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } }
I20260925 16:42:43.397562 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:43.397995 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6095675a9442d08651617b089590ea. Latest consensus state: current_term: 1 leader_uuid: "dd6095675a9442d08651617b089590ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } } }
I20260925 16:42:43.398118 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:43.398536 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6095675a9442d08651617b089590ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6095675a9442d08651617b089590ea" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 46827 } } }
I20260925 16:42:43.398682 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:43.398536 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:43.403275 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:43.404346 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:43.395642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54401 (local address 127.0.97.254:46827)
0925 16:42:43.395767 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:43.395771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:43.395950 (+ 179us) server_negotiation.cc:407] Connection header received
0925 16:42:43.396057 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:43.396060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:43.396108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:43.396205 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:43.397029 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:43.397825 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:43.399006 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:43.399354 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:43.399456 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:43.401229 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:43.401231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:43.401233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:43.401252 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:43.402748 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:43.403168 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:43.403173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:43.403174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:43.403250 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:43.403533 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:43.403539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:43.403542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:43.403710 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:43.403846 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:43.404074 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 16:42:43.404188 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:42:43.408867 22383 catalog_manager.cc:1347] Generated new cluster ID: 21aeed8829ed4920a23ea09908249d7f
I20260925 16:42:43.408939 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:43.440208 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:43.441200 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:43.445868 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6095675a9442d08651617b089590ea: Generated new TSK 0
I20260925 16:42:43.446650 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:43.461122 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22301
2026-09-25T16:42:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22400](info): set up 2 sockets
Sep 25 16:42:43 dist-test-slave-kxl8 krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260925 16:42:44.506901 391 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:44 dist-test-slave-kxl8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:44.628732 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39439
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:39439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:44.737126 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:44.737447 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:44.737532 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:44.741475 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:44.741571 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:44.741609 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:44.741644 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:44.741675 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:44.746759 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:39439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:39439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:44.747771 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:44.748756 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:44.757124 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:44.757190 22413 server_base.cc:1034] running on GCE node
W20260925 16:42:44.757129 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:44.757129 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:44.757967 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:44.758963 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:44.760146 22413 hybrid_clock.cc:648] HybridClock initialized: now 1790354564760131 us; error 25 us; skew 500 ppm
Sep 25 16:42:44 dist-test-slave-kxl8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:44.762939 22413 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:44.763804 22413 webserver.cc:460] Webserver started at http://127.0.97.254:33283/ using document root <none> and password file <none>
I20260925 16:42:44.764344 22413 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:44.764427 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:44.764635 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:44.766674 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "dfbc493de40c43bf9d41960a75616329"
format_stamp: "Formatted at 2026-09-25 16:42:44 on dist-test-slave-kxl8"
server_key: "76bd0fe4f451479affda8d6a3cc214b1"
server_key_iv: "db08bbb96fc2007fc84aba56fd72c78e"
server_key_version: "encryptionkey@0"
I20260925 16:42:44.767212 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "dfbc493de40c43bf9d41960a75616329"
format_stamp: "Formatted at 2026-09-25 16:42:44 on dist-test-slave-kxl8"
server_key: "76bd0fe4f451479affda8d6a3cc214b1"
server_key_iv: "db08bbb96fc2007fc84aba56fd72c78e"
server_key_version: "encryptionkey@0"
I20260925 16:42:44.770752 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:42:44.772967 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:44.774165 22413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:42:44.774297 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "dfbc493de40c43bf9d41960a75616329"
format_stamp: "Formatted at 2026-09-25 16:42:44 on dist-test-slave-kxl8"
server_key: "76bd0fe4f451479affda8d6a3cc214b1"
server_key_iv: "db08bbb96fc2007fc84aba56fd72c78e"
server_key_version: "encryptionkey@0"
I20260925 16:42:44.774400 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:44.793597 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:44.796905 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:44.797106 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:44.803841 22413 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:39439
I20260925 16:42:44.803890 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:39439 every 8 connection(s)
I20260925 16:42:44.804809 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:44.807510 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:44.812744 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329: Bootstrap starting.
I20260925 16:42:44.814843 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22413
I20260925 16:42:44.814981 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:44.815145 22481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:44.815244 391 external_mini_cluster.cc:1428] Setting key 5c9725cede7b6db0d5f0a74016e83e9b
I20260925 16:42:44.815886 22481 log.cc:824] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:44.817901 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329: No bootstrap required, opened a new log
I20260925 16:42:44.821990 22481 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } }
I20260925 16:42:44.822190 22481 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:44.822224 22481 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbc493de40c43bf9d41960a75616329, State: Initialized, Role: FOLLOWER
I20260925 16:42:44.822729 22481 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } }
I20260925 16:42:44.822896 22481 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:44.822964 22481 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:44.823060 22481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:44.823697 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:44.816278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40257 (local address 127.0.97.254:39439)
0925 16:42:44.816644 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:44.816664 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:44.816707 (+ 43us) server_negotiation.cc:407] Connection header received
0925 16:42:44.817347 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:44.817363 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:44.817673 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:44.817998 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:44.818347 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:44.819260 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:44.820515 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:44.820963 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:44.821083 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:44.821925 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:44.821930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:44.821942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:44.822412 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:44.822464 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:44.822488 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:44.822567 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:44.822701 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 16:42:44.822907 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260925 16:42:44.823957 22481 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } }
I20260925 16:42:44.824286 22481 leader_election.cc:302] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbc493de40c43bf9d41960a75616329; no voters:
I20260925 16:42:44.824571 22481 leader_election.cc:288] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:44.824680 22486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:44.824898 22486 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [term 1 LEADER]: Becoming Leader. State: Replica: dfbc493de40c43bf9d41960a75616329, State: Running, Role: LEADER
I20260925 16:42:44.825217 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } }
I20260925 16:42:44.825832 22481 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:44.826802 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:44.826853 22487 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbc493de40c43bf9d41960a75616329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } } }
I20260925 16:42:44.826948 22487 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:44.827203 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbc493de40c43bf9d41960a75616329. Latest consensus state: current_term: 1 leader_uuid: "dfbc493de40c43bf9d41960a75616329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbc493de40c43bf9d41960a75616329" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 39439 } } }
I20260925 16:42:44.827263 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:44.828248 22495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:44.831558 22495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:44.832691 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:44.829005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40283 (local address 127.0.97.254:39439)
0925 16:42:44.829106 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:44.829110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:44.829333 (+ 223us) server_negotiation.cc:407] Connection header received
0925 16:42:44.829491 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:44.829494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:44.829535 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:44.829620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:44.829861 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:44.830360 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:44.831136 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:44.831389 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:44.831495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:44.832282 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:44.832285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:44.832288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:44.832299 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:44.832323 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:44.832340 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:44.832414 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:44.832517 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 16:42:44.832562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:42:44.836997 22495 catalog_manager.cc:1347] Generated new cluster ID: 728209e80643443fa394f09d28762a0f
I20260925 16:42:44.837060 22495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:44.867765 22495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:44.868757 22495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:44.873128 22495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbc493de40c43bf9d41960a75616329: Generated new TSK 0
I20260925 16:42:44.873802 22495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:44.879465 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22413
2026-09-25T16:42:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:44 dist-test-slave-kxl8 krb5kdc[22512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:44 dist-test-slave-kxl8 krb5kdc[22512](info): set up 2 sockets
Sep 25 16:42:44 dist-test-slave-kxl8 krb5kdc[22512](info): commencing operation
krb5kdc: starting...
W20260925 16:42:45.917527 391 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:45 dist-test-slave-kxl8 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:46.035645 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:33727
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:33727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:46.143298 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:46.143532 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:46.143570 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:46.147445 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:46.147500 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:46.147518 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:46.147534 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:46.147549 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:46.152489 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:33727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:46.153491 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:46.154434 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:46.162460 22525 server_base.cc:1034] running on GCE node
W20260925 16:42:46.162397 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:46.162403 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:46.162411 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:46.163239 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:46.164295 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:46.165442 22525 hybrid_clock.cc:648] HybridClock initialized: now 1790354566165402 us; error 54 us; skew 500 ppm
Sep 25 16:42:46 dist-test-slave-kxl8 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:46.168143 22525 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:46.169016 22525 webserver.cc:460] Webserver started at http://127.0.97.254:38315/ using document root <none> and password file <none>
I20260925 16:42:46.169617 22525 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:46.169703 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:46.169956 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:46.171958 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "8ef7dd692b9b4b9e88ee90db19d04e47"
format_stamp: "Formatted at 2026-09-25 16:42:46 on dist-test-slave-kxl8"
server_key: "bd38af5bf06dab6fabee28b8e11f1bac"
server_key_iv: "a85da3dfa78f93e9cae5c1262251b067"
server_key_version: "encryptionkey@0"
I20260925 16:42:46.172513 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "8ef7dd692b9b4b9e88ee90db19d04e47"
format_stamp: "Formatted at 2026-09-25 16:42:46 on dist-test-slave-kxl8"
server_key: "bd38af5bf06dab6fabee28b8e11f1bac"
server_key_iv: "a85da3dfa78f93e9cae5c1262251b067"
server_key_version: "encryptionkey@0"
I20260925 16:42:46.176141 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:42:46.178591 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:46.179792 22525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:46.179922 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "8ef7dd692b9b4b9e88ee90db19d04e47"
format_stamp: "Formatted at 2026-09-25 16:42:46 on dist-test-slave-kxl8"
server_key: "bd38af5bf06dab6fabee28b8e11f1bac"
server_key_iv: "a85da3dfa78f93e9cae5c1262251b067"
server_key_version: "encryptionkey@0"
I20260925 16:42:46.180056 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:46.184487 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:46.187546 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:46.187726 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:46.194738 22525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:33727
I20260925 16:42:46.194758 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:33727 every 8 connection(s)
I20260925 16:42:46.195655 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:46.198523 22593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:46.200927 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22525
I20260925 16:42:46.201062 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:46.201412 391 external_mini_cluster.cc:1428] Setting key 97128571da47814581c40292cb353186
I20260925 16:42:46.204574 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47: Bootstrap starting.
I20260925 16:42:46.206455 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:46.202805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56177 (local address 127.0.97.254:33727)
0925 16:42:46.203235 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:46.203250 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:46.203283 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:42:46.203898 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:46.203913 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:46.204210 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:46.204539 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:46.205325 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:46.205331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:46.205350 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:46.205361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:46.205411 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:46.205430 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:46.205571 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:46.205640 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 16:42:46.205788 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260925 16:42:46.207046 22593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:46.207819 22593 log.cc:824] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:46.209110 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:42:46.210064 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47: No bootstrap required, opened a new log
I20260925 16:42:46.212828 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:46.211346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52029 (local address 127.0.97.254:33727)
0925 16:42:46.211466 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:46.211470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:46.211614 (+ 144us) server_negotiation.cc:407] Connection header received
0925 16:42:46.211707 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:46.211709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:46.211743 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:46.211818 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:46.212458 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:46.212462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:46.212464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:42:46.212476 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:46.212502 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:42:46.212514 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:46.212596 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:46.212672 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 16:42:46.212709 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:42:46.214349 22593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } }
I20260925 16:42:46.214565 22593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:42:46.214598 22593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef7dd692b9b4b9e88ee90db19d04e47, State: Initialized, Role: FOLLOWER
I20260925 16:42:46.214942 22593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } }
I20260925 16:42:46.215050 22593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:42:46.215094 22593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:42:46.215160 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:46.215986 22593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } }
I20260925 16:42:46.216264 22593 leader_election.cc:302] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ef7dd692b9b4b9e88ee90db19d04e47; no voters:
I20260925 16:42:46.216511 22593 leader_election.cc:288] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:42:46.216605 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:46.216822 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef7dd692b9b4b9e88ee90db19d04e47, State: Running, Role: LEADER
I20260925 16:42:46.217145 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } }
I20260925 16:42:46.217703 22593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:46.218674 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } } }
I20260925 16:42:46.218803 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:46.219146 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ef7dd692b9b4b9e88ee90db19d04e47. Latest consensus state: current_term: 1 leader_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef7dd692b9b4b9e88ee90db19d04e47" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 33727 } } }
I20260925 16:42:46.219238 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:46.219949 22612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:46.222523 22612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:46.227994 22612 catalog_manager.cc:1347] Generated new cluster ID: 7f18388c657d41bda34e7b719b0ea2ec
I20260925 16:42:46.228060 22612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:46.242756 22612 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:46.243063 22612 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:46.252836 22612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ef7dd692b9b4b9e88ee90db19d04e47: Generated new TSK 0
I20260925 16:42:46.253533 22612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:42:46.258951 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22525
2026-09-25T16:42:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1379 ms)
[----------] 18 tests from AuthTokenIssuingTest (25274 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:46 dist-test-slave-kxl8 krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:46 dist-test-slave-kxl8 krb5kdc[22624](info): set up 2 sockets
Sep 25 16:42:46 dist-test-slave-kxl8 krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260925 16:42:47.301611 391 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:47.422325 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34057
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:47.530498 22637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:47.530838 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:47.530903 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:47.530985 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:47.534988 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:47.535065 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:47.535104 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:47.535156 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:47.535197 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:47.540334 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:47.541493 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:47.542492 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:47.550747 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:47.550761 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:47.550766 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:47.550808 22637 server_base.cc:1034] running on GCE node
I20260925 16:42:47.551542 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:47.552591 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:47.553762 22637 hybrid_clock.cc:648] HybridClock initialized: now 1790354567553739 us; error 41 us; skew 500 ppm
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:47.556344 22637 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:47.557217 22637 webserver.cc:460] Webserver started at http://127.0.97.254:33991/ using document root <none> and password file <none>
I20260925 16:42:47.557775 22637 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:47.557859 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:47.558079 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:47.559990 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "cf2ff74cf81744728d3fd15be812eca2"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "588d40110b80c851d58681d425d31e36"
server_key_iv: "9ece089e6358cd7b319d70bc3413a5f2"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.560536 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "cf2ff74cf81744728d3fd15be812eca2"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "588d40110b80c851d58681d425d31e36"
server_key_iv: "9ece089e6358cd7b319d70bc3413a5f2"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.564013 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:42:47.566269 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:47.567334 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:47.567463 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "cf2ff74cf81744728d3fd15be812eca2"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "588d40110b80c851d58681d425d31e36"
server_key_iv: "9ece089e6358cd7b319d70bc3413a5f2"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.567571 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:47.610764 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:47.614118 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:47.614327 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:47.621373 22637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:34057
I20260925 16:42:47.621652 22704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:34057 every 8 connection(s)
I20260925 16:42:47.622378 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:47.625099 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:47.628120 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22637
I20260925 16:42:47.628262 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:47.628571 391 external_mini_cluster.cc:1428] Setting key 72a76a3b21aae27bffacabfe0ff9341c
I20260925 16:42:47.629411 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.253@KRBTEST.COM" created.
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:47.645810 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34057 (local address 127.0.0.1:51656)
0925 16:42:47.632607 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.632668 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:47.633570 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:47.634292 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:47.634301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:47.634788 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:47.635175 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.635194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.636352 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.636356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.637474 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.637483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.638141 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.638144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.638276 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.638854 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:47.638871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:47.640355 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:47.642567 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.642579 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.642593 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.642908 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.643326 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.643332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.643337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.643563 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.644168 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:47.644179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:47.644366 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 16:42:47.644599 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 16:42:47.645020 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":105,"spinlock_wait_cycles":42752,"thread_start_us":139,"threads_started":1}
I20260925 16:42:47.645808 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.97.254:34057)
0925 16:42:47.632473 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:47.632491 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:47.633792 (+ 1301us) server_negotiation.cc:407] Connection header received
0925 16:42:47.633895 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:47.633898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:47.634073 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:47.634208 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:47.635353 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.636221 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.637594 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.637992 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.638109 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.640507 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:47.640516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:47.640533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:47.640565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:47.642405 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.643013 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.643017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.643018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.643121 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.643681 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.643684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.643685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.644048 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:47.644148 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:47.644644 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 16:42:47.644913 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":279,"thread_start_us":150,"threads_started":1}
I20260925 16:42:47.648746 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:47.649523 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.649222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42554)
0925 16:42:47.649375 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.649456 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 16:42:47.649775 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
I20260925 16:42:47.650583 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.650391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42564)
0925 16:42:47.650493 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.650534 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:42:47.650792 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:45621: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
I20260925 16:42:47.656880 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:45621
--webserver_interface=127.0.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:42:47.700057 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } attempt: 1
I20260925 16:42:47.700959 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.700644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42566)
0925 16:42:47.700798 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.700871 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:42:47.701725 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.701511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42578)
0925 16:42:47.701613 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.701663 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 16:42:47.701855 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:45621: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
W20260925 16:42:47.766746 22712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:47.767022 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:47.767046 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:47.767094 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 16:42:47.770001 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } attempt: 2
I20260925 16:42:47.770812 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.770507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42592)
0925 16:42:47.770657 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.770726 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:42:47.771379 22712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:47.771426 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:47.771464 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:47.771541 22712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:47.771581 22712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:47.771685 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.771405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42606)
0925 16:42:47.771571 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.771625 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 16:42:47.771817 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:45621: Network error: Client connection negotiation failed: client connection to 127.0.97.253:45621: connect: Connection refused (error 111)
I20260925 16:42:47.776924 22712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:47.778026 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:47.778954 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:47.787014 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:47.787107 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:47.787096 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:47.787197 22712 server_base.cc:1034] running on GCE node
I20260925 16:42:47.787825 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:47.788846 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:47.789999 22712 hybrid_clock.cc:648] HybridClock initialized: now 1790354567789984 us; error 25 us; skew 500 ppm
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:47.792625 22712 init.cc:377] Logged in from keytab as kudu/127.0.97.253@KRBTEST.COM (short username kudu)
I20260925 16:42:47.793526 22712 webserver.cc:460] Webserver started at http://127.0.97.253:44985/ using document root <none> and password file <none>
I20260925 16:42:47.794060 22712 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:47.794118 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:47.794337 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:47.796289 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data/instance:
uuid: "9abcc2ad04064ef78bda26cc735e468e"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "594cfca8d47707afc684c6d9fa658d70"
server_key_iv: "3e3efe955346015a897fac6684e344ea"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.796831 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal/instance:
uuid: "9abcc2ad04064ef78bda26cc735e468e"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "594cfca8d47707afc684c6d9fa658d70"
server_key_iv: "3e3efe955346015a897fac6684e344ea"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.800446 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:42:47.802754 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:47.803881 22712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:42:47.804001 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "9abcc2ad04064ef78bda26cc735e468e"
format_stamp: "Formatted at 2026-09-25 16:42:47 on dist-test-slave-kxl8"
server_key: "594cfca8d47707afc684c6d9fa658d70"
server_key_iv: "3e3efe955346015a897fac6684e344ea"
server_key_version: "encryptionkey@0"
I20260925 16:42:47.804104 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:47.828902 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:47.831748 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:47.831943 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:47.838680 22712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.253:45621
I20260925 16:42:47.838943 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.253:45621 every 8 connection(s)
I20260925 16:42:47.839609 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
I20260925 16:42:47.842245 22780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:47.842679 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22712
I20260925 16:42:47.842818 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-1/wal/instance
I20260925 16:42:47.843077 391 external_mini_cluster.cc:1428] Setting key 7366d682fe5d2d85ecaeecf3d04fa75a
I20260925 16:42:47.846932 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.252@KRBTEST.COM" created.
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:47.861181 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.849366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.0.97.254:34057)
0925 16:42:47.849520 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:47.849525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:47.850408 (+ 883us) server_negotiation.cc:407] Connection header received
0925 16:42:47.851381 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:47.851385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:47.851436 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:47.851526 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:47.852674 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.853212 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.854542 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.854978 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.855105 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.857734 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:47.857738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:47.857739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:47.857764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:47.859209 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.859700 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.859705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.859707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.859784 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.860121 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.860124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.860126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.860300 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:47.860403 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:47.860884 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 16:42:47.861006 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:42:47.862022 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.849769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34057 (local address 127.0.0.1:51662)
0925 16:42:47.850231 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.850282 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:47.851163 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:47.851685 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:47.851692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:47.852072 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:47.852539 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.852552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.853370 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.853372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.854406 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.854415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.855116 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.855119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.855311 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.855936 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:47.855961 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:47.857562 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:47.859335 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.859340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.859354 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.859603 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.859878 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.859881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.859883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.860023 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.860401 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:47.860407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:47.860649 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 16:42:47.860830 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 16:42:47.861146 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260925 16:42:47.862941 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } attempt: 3
I20260925 16:42:47.863487 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
I20260925 16:42:47.868841 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:43981
--webserver_interface=127.0.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
Sep 25 16:42:47 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:42:47.874647 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.863916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42614)
0925 16:42:47.864025 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.864039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:47.864111 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:47.864473 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:47.864475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:47.864639 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:47.864719 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.864723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.865837 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.865841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.866722 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.866731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.867641 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.867644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.867750 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.869037 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:47.869057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:47.870433 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:47.872572 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.872578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.872581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.872837 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.873467 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.873470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.873591 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.874089 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:47.874092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:47.874223 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 16:42:47.874331 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 16:42:47.874463 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":29,"spinlock_wait_cycles":5632}
I20260925 16:42:47.874671 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.864049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.0.97.253:45621)
0925 16:42:47.864303 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:47.864312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:47.864324 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:42:47.864351 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:47.864353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:47.864390 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:47.864453 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:47.864820 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.865748 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.866822 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.867502 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.868113 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.870602 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:47.870608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:47.870612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:47.870640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:47.872464 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.872938 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.872944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.872946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.873024 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873667 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.873672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.873674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.873984 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:47.874120 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:47.874411 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 16:42:47.874544 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":647,"server-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260925 16:42:47.875020 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.863360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42608)
0925 16:42:47.863523 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.863538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:47.863618 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:47.864212 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:47.864215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:47.864381 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:47.864479 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.864485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.865837 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.865841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.866699 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:47.866709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.867640 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.867643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:47.867769 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.868365 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:47.868379 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:47.869938 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:47.873118 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.873125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.873373 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.873762 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:47.873767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:47.873852 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:47.874383 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:47.874389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:47.874510 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 16:42:47.874608 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 16:42:47.874766 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 16:42:47.876011 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.863503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.0.97.253:45621)
0925 16:42:47.863832 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:47.863841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:47.863860 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:42:47.863917 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:47.863925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:47.864090 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:47.864231 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:47.864593 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.865718 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.866822 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:47.867483 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:47.867614 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:47.870104 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:47.870118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:47.870137 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:47.870167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:47.873006 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873466 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.873472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.873477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.873613 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:47.873964 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:47.873970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:47.873974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:47.874249 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:47.875469 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:47.875556 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 16:42:47.875701 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260925 16:42:47.876905 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:47.876945 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:47.877619 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.877383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43288)
0925 16:42:47.877493 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.877569 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 16:42:47.877813 22729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
I20260925 16:42:47.878260 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.877433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43294)
0925 16:42:47.878115 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.878177 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597}
I20260925 16:42:47.878610 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.878399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43302)
0925 16:42:47.878507 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.878553 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:42:47.878952 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
I20260925 16:42:47.879107 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.878791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43316)
0925 16:42:47.878972 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.879033 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 16:42:47.879230 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
I20260925 16:42:47.910383 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 1
I20260925 16:42:47.911275 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.910917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43322)
0925 16:42:47.911085 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.911173 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 16:42:47.912024 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.911796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43338)
0925 16:42:47.911898 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.911959 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 16:42:47.912163 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
I20260925 16:42:47.928212 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 1
I20260925 16:42:47.928970 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.928711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43354)
0925 16:42:47.928837 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.928899 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:42:47.929658 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.929452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43364)
0925 16:42:47.929559 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.929608 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:42:47.929778 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
W20260925 16:42:47.981242 22787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:47.981578 22787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:47.981653 22787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:47.981729 22787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:47.985677 22787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:47.985755 22787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:47.985791 22787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:47.985841 22787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:47.985872 22787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:47.987331 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 2
I20260925 16:42:47.988147 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.987812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43372)
0925 16:42:47.987985 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.988054 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 16:42:47.988821 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.988616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43374)
0925 16:42:47.988716 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.988764 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 16:42:47.988947 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
I20260925 16:42:47.991552 22787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:34057,127.0.97.253:45621,127.0.97.252:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:43981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:47.992522 22787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:47.993557 22787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:47.997910 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 2
I20260925 16:42:47.998593 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.998341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43384)
0925 16:42:47.998460 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.998527 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:42:47.999218 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:47.999022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43400)
0925 16:42:47.999120 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:47.999164 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 16:42:47.999320 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:43981: Network error: Client connection negotiation failed: client connection to 127.0.97.252:43981: connect: Connection refused (error 111)
W20260925 16:42:48.001770 22793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:48.001773 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:48.001773 22796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:48.002066 22787 server_base.cc:1034] running on GCE node
I20260925 16:42:48.002574 22787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:48.003624 22787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:48.004796 22787 hybrid_clock.cc:648] HybridClock initialized: now 1790354568004769 us; error 45 us; skew 500 ppm
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:48.007551 22787 init.cc:377] Logged in from keytab as kudu/127.0.97.252@KRBTEST.COM (short username kudu)
I20260925 16:42:48.008446 22787 webserver.cc:460] Webserver started at http://127.0.97.252:43775/ using document root <none> and password file <none>
I20260925 16:42:48.008991 22787 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:48.009073 22787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:48.009266 22787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:48.011252 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data/instance:
uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46"
format_stamp: "Formatted at 2026-09-25 16:42:48 on dist-test-slave-kxl8"
server_key: "1cf4e992dc9873d02c24620a4ba5b987"
server_key_iv: "9142a2913428b3a7d5370de7d85127f3"
server_key_version: "encryptionkey@0"
I20260925 16:42:48.011796 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal/instance:
uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46"
format_stamp: "Formatted at 2026-09-25 16:42:48 on dist-test-slave-kxl8"
server_key: "1cf4e992dc9873d02c24620a4ba5b987"
server_key_iv: "9142a2913428b3a7d5370de7d85127f3"
server_key_version: "encryptionkey@0"
I20260925 16:42:48.015300 22787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:42:48.017627 22803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:48.018757 22787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:42:48.018883 22787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46"
format_stamp: "Formatted at 2026-09-25 16:42:48 on dist-test-slave-kxl8"
server_key: "1cf4e992dc9873d02c24620a4ba5b987"
server_key_iv: "9142a2913428b3a7d5370de7d85127f3"
server_key_version: "encryptionkey@0"
I20260925 16:42:48.018982 22787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:48.029193 22787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:48.031751 22787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:48.031940 22787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:48.038671 22787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.252:43981
I20260925 16:42:48.038926 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.252:43981 every 8 connection(s)
I20260925 16:42:48.039631 22787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
I20260925 16:42:48.042335 22856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:48.045903 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:48.046208 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 22787
I20260925 16:42:48.046347 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790354288967115-391-0/minicluster-data/master-2/wal/instance
I20260925 16:42:48.046615 391 external_mini_cluster.cc:1428] Setting key 36dec3b8f6b259fa060e4820618f93ad
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:42:48.058595 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.047776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.0.97.254:34057)
0925 16:42:48.047894 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.047898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.048466 (+ 568us) server_negotiation.cc:407] Connection header received
0925 16:42:48.049224 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.049227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.049319 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.049417 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.050336 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.050822 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.051900 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.052157 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.052233 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.054811 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.054815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.054817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.054839 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.056513 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.057081 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.057086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.057090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.057189 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.057560 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.057563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.057569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.057763 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.057865 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.058299 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.058412 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:42:48.059473 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.047867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:34057 (local address 127.0.0.1:51674)
0925 16:42:48.048339 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:48.048373 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:48.049040 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:48.049474 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:48.049479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:48.049904 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:48.050192 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.050206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.050932 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.050935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.051789 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.051798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.052283 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.052287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.052497 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.053245 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:48.053260 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:48.054674 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:48.056632 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.056640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.056646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.056976 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.057304 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.057309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.057311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.057438 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.057865 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:48.057872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:48.058097 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 16:42:48.058259 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 16:42:48.058544 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":62,"spinlock_wait_cycles":78976,"thread_start_us":170,"threads_started":1}
I20260925 16:42:48.059477 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.047868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.0.97.252:43981)
0925 16:42:48.048265 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.048281 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.048319 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:42:48.049031 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.049047 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.049228 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.049443 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.050154 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.051144 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.051983 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.052416 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.052532 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.055249 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.055258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.055274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.055293 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.057022 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.057463 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.057469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.057471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.057560 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.057841 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.057845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.057847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.058089 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.058283 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.058498 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.058632 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":276,"spinlock_wait_cycles":7040,"thread_start_us":120,"threads_started":1}
I20260925 16:42:48.060554 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:48.069322 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.061008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:45621 (local address 127.0.0.1:42618)
0925 16:42:48.061171 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:48.061186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:48.061252 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:48.061527 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:48.061529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:48.061710 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:48.061817 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.061825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.062860 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.062863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.063694 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.063705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.064262 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.064264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.064333 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.064774 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:48.064785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:48.066210 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:48.067925 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.067928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.067930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.068114 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.068388 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.068391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.068392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.068468 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.068829 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:48.068832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:48.068923 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 16:42:48.069025 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:42:48.069142 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 16:42:48.069367 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.061119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.0.97.253:45621)
0925 16:42:48.061239 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.061245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.061259 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:42:48.061369 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.061372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.061419 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.061505 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.061945 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.062742 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.063809 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.064157 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.064249 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.066339 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.066342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.066344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.066366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.067818 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.068213 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.068218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.068220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.068294 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.068567 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.068572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.068574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.068739 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.068839 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.069075 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.069187 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:42:48.069993 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:48.071095 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.062014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.0.97.254:34057)
0925 16:42:48.062120 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.062124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.062135 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:42:48.062167 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.062169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.062198 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.062255 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.062932 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.063658 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.064621 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.064837 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.064937 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.067590 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.067593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.067595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.067615 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.068957 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.069776 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.069779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.069780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.069876 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.070242 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.070248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.070251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.070472 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.070685 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.070833 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.070941 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:42:48.078174 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.070406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43422)
0925 16:42:48.070505 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:48.070519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:48.070582 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:48.070805 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:48.070808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:48.070960 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:48.071037 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.071064 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.071907 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.071910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.072708 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.072720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.073170 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.073173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.073266 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.073713 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:48.073722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:48.074835 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:48.076632 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.076634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.076636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.076809 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.077104 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.077107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.077109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.077193 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.077649 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:48.077652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:48.077723 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 16:42:48.077804 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:42:48.077962 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":19}
I20260925 16:42:48.078274 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.070504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.97.252:43981)
0925 16:42:48.070615 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.070619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.070630 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:42:48.070660 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.070663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.070701 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.070780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.071172 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.071814 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.072822 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.073060 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.073161 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.074950 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.074954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.074956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.074982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.076511 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.076899 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.076903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.076905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.076992 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.077362 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.077364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.077365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.077546 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.077643 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.077883 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.078042 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":60}
I20260925 16:42:48.081308 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.072202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.97.253:45621)
0925 16:42:48.072294 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.072298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.072316 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:42:48.072360 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.072362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.072391 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.072444 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.073095 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.073812 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.074641 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.074868 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.074968 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.077580 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.077587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.077590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.077618 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.079126 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.079612 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.079615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.079617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.079705 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.080005 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.080008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.080009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.080195 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.080375 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.080969 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.081138 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 16:42:48.082656 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: Bootstrap starting.
I20260925 16:42:48.085474 22856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:48.086374 22856 log.cc:824] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:48.089013 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 3
I20260925 16:42:48.090402 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } attempt: 3
I20260925 16:42:48.092237 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: No bootstrap required, opened a new log
I20260925 16:42:48.092293 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:42:48.093540 22679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cf2ff74cf81744728d3fd15be812eca2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51690
Sep 25 16:42:48 dist-test-slave-kxl8 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:42:48.097183 22856 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.097514 22856 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7a4f99ad1a4dadb398bb01bb1dbb46, State: Initialized, Role: FOLLOWER
I20260925 16:42:48.097898 22856 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.098416 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.098650 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.098815 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.089706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43436)
0925 16:42:48.089956 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:48.089970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:48.090049 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:48.090285 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:48.090288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:48.090474 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:48.090584 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.090589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.091581 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.091584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.092318 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.092329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.092973 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.092976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.093052 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.093575 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:48.093590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:48.095138 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:48.097048 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.097052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.097057 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.097324 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.097665 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.097668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.097670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.097763 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.098240 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:48.098243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:48.098398 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 16:42:48.098525 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 16:42:48.098658 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 16:42:48.098869 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.089938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.97.252:43981)
0925 16:42:48.090033 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.090037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.090052 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:42:48.090134 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.090137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.090175 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.090255 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.090714 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.091433 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.092447 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.092768 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.092862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.095282 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.095285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.095288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.095313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.096935 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.097442 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.097448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.097450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.097543 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.097857 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.097863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.097868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.098147 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.098269 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.098608 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.098716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 16:42:48.099504 22856 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:48.102432 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: Bootstrap starting.
I20260925 16:42:48.102650 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.090756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:43981 (local address 127.0.0.1:43444)
0925 16:42:48.090916 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:48.090930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:48.091010 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:48.091856 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:48.091860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:48.092033 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:48.092134 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.092141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.093213 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.093217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.094016 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:48.094025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.094659 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.094662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:48.094744 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.095560 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:48.095573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:48.096945 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:48.100869 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.100883 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.100885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.101138 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.101550 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:48.101556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:48.101559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:48.101658 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:48.102136 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:48.102139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:48.102235 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 16:42:48.102341 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:42:48.102478 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 16:42:48.105247 22705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:48.106096 22705 log.cc:824] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:48.102870 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.091184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.0.97.252:43981)
0925 16:42:48.091623 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.091634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.091647 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:42:48.091679 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.091685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.091732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.092240 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.092371 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.093082 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.094158 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.094530 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.094940 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.097485 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.097489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.097491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.097515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.100727 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.101306 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.101309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.101311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.101412 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.101779 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.101784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.101789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.102005 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.102143 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.102453 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.102625 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":89,"threads_started":1}
W20260925 16:42:48.105849 22874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:48.107613 22874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:48.109175 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: No bootstrap required, opened a new log
I20260925 16:42:48.110242 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: Bootstrap starting.
I20260925 16:42:48.108523 22787 master_runner.cc:186] Error getting master registration for 127.0.97.254:34057: OK, instance_id { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" instance_seqno: 1790354567619483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:42:48.111865 22787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:42:48.112114 22705 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.112306 22705 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2ff74cf81744728d3fd15be812eca2, State: Initialized, Role: FOLLOWER
I20260925 16:42:48.112598 22705 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.112722 22780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:48.113142 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.113338 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.113544 22780 log.cc:824] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:48.113920 22705 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:48.114037 22679 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:42:48.114184 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:48.115411 22679 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.116379 22679 leader_election.cc:288] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [CANDIDATE]: Term 1 election: Requested vote from peers 9abcc2ad04064ef78bda26cc735e468e (127.0.97.253:45621), bf7a4f99ad1a4dadb398bb01bb1dbb46 (127.0.97.252:43981)
W20260925 16:42:48.116919 22658 tablet.cc:2369] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:48.116977 22830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf2ff74cf81744728d3fd15be812eca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46"
I20260925 16:42:48.117230 22830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:48.118497 22830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2ff74cf81744728d3fd15be812eca2 in term 1.
I20260925 16:42:48.118943 22654 leader_election.cc:302] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7a4f99ad1a4dadb398bb01bb1dbb46, cf2ff74cf81744728d3fd15be812eca2; no voters:
I20260925 16:42:48.119225 22877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:48.119294 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: No bootstrap required, opened a new log
I20260925 16:42:48.119407 22877 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [term 1 LEADER]: Becoming Leader. State: Replica: cf2ff74cf81744728d3fd15be812eca2, State: Running, Role: LEADER
I20260925 16:42:48.119700 22877 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.121323 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ff74cf81744728d3fd15be812eca2. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.121454 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:48.118431 22637 master_runner.cc:186] Error getting master registration for 127.0.97.253:45621: OK, instance_id { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" instance_seqno: 1790354567836770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:42:48.121709 22637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:42:48.123347 22780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.123579 22780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abcc2ad04064ef78bda26cc735e468e, State: Initialized, Role: FOLLOWER
I20260925 16:42:48.123893 22780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } }
I20260925 16:42:48.124855 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.125134 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:42:48.126078 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:48.126153 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:48.125200 22780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:48.125293 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf2ff74cf81744728d3fd15be812eca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9abcc2ad04064ef78bda26cc735e468e"
I20260925 16:42:48.126366 22753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:48.127697 22753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2ff74cf81744728d3fd15be812eca2 in term 1.
W20260925 16:42:48.132745 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:48.132886 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:48.133405 22897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:48.134416 22897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:42:48.139277 22830 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [term 1 FOLLOWER]: Refusing update from remote peer cf2ff74cf81744728d3fd15be812eca2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:48.139349 22753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [term 1 FOLLOWER]: Refusing update from remote peer cf2ff74cf81744728d3fd15be812eca2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:48.139881 22891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:48.140039 22877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:48.143498 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ff74cf81744728d3fd15be812eca2. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.143610 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.144286 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ff74cf81744728d3fd15be812eca2. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.144403 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.147491 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.147624 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:48.147809 22914 mvcc.cc:205] Tried to move back new op lower bound from 7333292311092535296 to 7333292311021309952. Current Snapshot: MvccSnapshot[applied={T|T < 7333292311092535296}]
I20260925 16:42:48.147840 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.147957 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.148303 22915 mvcc.cc:205] Tried to move back new op lower bound from 7333292311092535296 to 7333292311021309952. Current Snapshot: MvccSnapshot[applied={T|T < 7333292311092535296}]
I20260925 16:42:48.152873 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.130853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.97.253:45621)
0925 16:42:48.130956 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.130960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.141690 (+ 10730us) server_negotiation.cc:407] Connection header received
0925 16:42:48.141876 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.141880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.141926 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.141977 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.143127 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.143858 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.145248 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.145601 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.145698 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.149032 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.149036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.149038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.149063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.151070 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.151563 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.151566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.151568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.151653 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.151945 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.151948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.151950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.152157 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.152256 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.152602 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.152715 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:42:48.158495 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.130494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.0.97.254:34057)
0925 16:42:48.130709 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.130716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.145449 (+ 14733us) server_negotiation.cc:407] Connection header received
0925 16:42:48.146979 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.146982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.147035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.147084 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.147894 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.148660 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.149867 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.150257 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.150344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.154603 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.154609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.154614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.154646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.156418 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.156947 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.156952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.156955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.157063 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.157435 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.157440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.157445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.157681 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.157813 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.158187 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.158314 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 16:42:48.159795 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:48.131918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.97.252:43981)
0925 16:42:48.132045 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.132050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.145651 (+ 13601us) server_negotiation.cc:407] Connection header received
0925 16:42:48.145909 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.145913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.145960 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.146013 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.150520 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.151312 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.152404 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:48.152770 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:48.152841 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:48.155921 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:48.155927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:48.155933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:48.155963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:48.157778 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.158243 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.158247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.158248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.158331 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:48.158655 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:48.158658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:48.158660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:48.158872 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:48.158982 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:48.159432 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 16:42:48.159652 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:42:48.162896 22897 catalog_manager.cc:1347] Generated new cluster ID: ee955a7ec7904612a959745038be0563
I20260925 16:42:48.162962 22897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:48.171556 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.171674 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:48.172706 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf2ff74cf81744728d3fd15be812eca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ff74cf81744728d3fd15be812eca2" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 34057 } } peers { permanent_uuid: "9abcc2ad04064ef78bda26cc735e468e" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 45621 } } peers { permanent_uuid: "bf7a4f99ad1a4dadb398bb01bb1dbb46" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 43981 } } }
I20260925 16:42:48.172806 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:48.173286 22916 mvcc.cc:205] Tried to move back new op lower bound from 7333292311092535296 to 7333292311021309952. Current Snapshot: MvccSnapshot[applied={T|T < 7333292311092535296}]
I20260925 16:42:48.193843 22897 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:48.195276 22897 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:48.206113 22897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2ff74cf81744728d3fd15be812eca2: Generated new TSK 0
I20260925 16:42:48.207073 22897 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:42:48.237113 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.234164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.97.253:45621)
0925 16:42:48.234297 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.234301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.236035 (+ 1734us) server_negotiation.cc:407] Connection header received
0925 16:42:48.236228 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.236232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.236281 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.236333 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.236980 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42654: BlockingRecv error: recv got EOF from 127.0.0.1:42654 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 16:42:48.238503 22708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.234946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.97.254:34057)
0925 16:42:48.235054 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.235058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.237609 (+ 2551us) server_negotiation.cc:407] Connection header received
0925 16:42:48.237776 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.237779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.237829 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.237881 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.238367 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 16:42:48.239867 22862 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.235536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.0.97.252:43981)
0925 16:42:48.235668 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.235673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.238871 (+ 3198us) server_negotiation.cc:407] Connection header received
0925 16:42:48.239187 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.239190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.239240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.239290 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.239757 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43462: BlockingRecv error: recv got EOF from 127.0.0.1:43462 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 16:42:48.244161 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.242068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.253:45621)
0925 16:42:48.242162 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.242166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.243244 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 16:42:48.243469 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.243473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.243549 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.243601 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.244063 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42658: BlockingRecv error: recv got EOF from 127.0.0.1:42658 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 16:42:48.249150 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.247618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.97.253:45621)
0925 16:42:48.247719 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.247723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.248258 (+ 535us) server_negotiation.cc:407] Connection header received
0925 16:42:48.248459 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.248463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.248515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.248567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.249050 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42674: BlockingRecv error: recv got EOF from 127.0.0.1:42674 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 16:42:48.254824 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.253192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.97.253:45621)
0925 16:42:48.253352 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.253356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.253930 (+ 574us) server_negotiation.cc:407] Connection header received
0925 16:42:48.254113 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.254116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.254166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.254219 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.254724 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42688: BlockingRecv error: recv got EOF from 127.0.0.1:42688 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:42:48.262408 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.260790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.97.253:45621)
0925 16:42:48.260931 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.260935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.261466 (+ 531us) server_negotiation.cc:407] Connection header received
0925 16:42:48.261669 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.261672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.261721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.261776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.262289 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42704: BlockingRecv error: recv got EOF from 127.0.0.1:42704 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 16:42:48.274506 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.272666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.0.97.253:45621)
0925 16:42:48.272796 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.272801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.273485 (+ 684us) server_negotiation.cc:407] Connection header received
0925 16:42:48.273701 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.273706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.273763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.273822 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.274391 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42716: BlockingRecv error: recv got EOF from 127.0.0.1:42716 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 16:42:48.294816 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.292872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.97.253:45621)
0925 16:42:48.293113 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.293118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.293746 (+ 628us) server_negotiation.cc:407] Connection header received
0925 16:42:48.293954 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.293958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.294012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.294069 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.294666 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42720: BlockingRecv error: recv got EOF from 127.0.0.1:42720 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 16:42:48.330511 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.329147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.97.253:45621)
0925 16:42:48.329308 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.329314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.329595 (+ 281us) server_negotiation.cc:407] Connection header received
0925 16:42:48.329763 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.329767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.329822 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.329907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.330397 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42722: BlockingRecv error: recv got EOF from 127.0.0.1:42722 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 16:42:48.397652 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.396490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.0.97.253:45621)
0925 16:42:48.396640 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.396643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.396719 (+ 76us) server_negotiation.cc:407] Connection header received
0925 16:42:48.396841 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.396844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.396890 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.396967 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.397527 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42738: BlockingRecv error: recv got EOF from 127.0.0.1:42738 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 16:42:48.528726 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.527504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.0.97.253:45621)
0925 16:42:48.527665 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.527670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.527842 (+ 172us) server_negotiation.cc:407] Connection header received
0925 16:42:48.528006 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.528009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.528069 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.528155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.528602 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42742: BlockingRecv error: recv got EOF from 127.0.0.1:42742 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 16:42:48.787935 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:48.786833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.0.97.253:45621)
0925 16:42:48.786964 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:48.786970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:48.787078 (+ 108us) server_negotiation.cc:407] Connection header received
0925 16:42:48.787241 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:48.787245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:48.787299 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:48.787387 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:48.787821 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42744: BlockingRecv error: recv got EOF from 127.0.0.1:42744 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:42:49.109014 22874 catalog_manager.cc:1259] Loaded cluster ID: ee955a7ec7904612a959745038be0563
I20260925 16:42:49.109089 22874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: loading cluster ID for follower catalog manager: success
I20260925 16:42:49.110692 22874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: acquiring CA information for follower catalog manager: success
I20260925 16:42:49.111789 22874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf7a4f99ad1a4dadb398bb01bb1dbb46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:42:49.134140 22909 catalog_manager.cc:1259] Loaded cluster ID: ee955a7ec7904612a959745038be0563
I20260925 16:42:49.134223 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: loading cluster ID for follower catalog manager: success
I20260925 16:42:49.135851 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: acquiring CA information for follower catalog manager: success
I20260925 16:42:49.136986 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9abcc2ad04064ef78bda26cc735e468e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:42:49.305655 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:49.302046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.97.253:45621)
0925 16:42:49.302273 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:49.302277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:49.302292 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:42:49.302414 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:49.302416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:49.302549 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:49.302660 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:49.302891 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.303494 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.304309 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.304525 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.304599 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:49.304847 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:49.305329 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:49.305434 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:49.305482 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 16:42:49.305531 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260925 16:42:49.312446 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:49.308725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.0.97.252:43981)
0925 16:42:49.308982 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:49.308986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:49.309000 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:42:49.309176 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:49.309179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:49.309308 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:49.309438 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:49.309658 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.310165 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.310984 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.311218 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.311324 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:49.311625 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:49.312041 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:49.312170 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:49.312231 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 16:42:49.312286 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260925 16:42:49.319705 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:49.315989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.97.252:43981)
0925 16:42:49.316084 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:49.316088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:49.316100 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:42:49.316164 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:49.316170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:49.316219 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:49.316279 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:49.316657 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.317396 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318519 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.318843 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318945 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:49.319212 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:49.319291 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:49.319359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:49.319508 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 16:42:49.319567 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:42:49.319728 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:49.315421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.97.253:45621)
0925 16:42:49.315570 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:49.315574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:49.315938 (+ 364us) server_negotiation.cc:407] Connection header received
0925 16:42:49.316150 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:49.316153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:49.316214 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:49.316283 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:49.316638 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.317139 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318442 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.318769 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318876 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:49.319279 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:49.319359 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:49.319426 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:49.319527 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 16:42:49.319577 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:42:49.319871 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:49.315369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.0.97.254:34057)
0925 16:42:49.315705 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:49.315708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:49.315737 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:42:49.315920 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:49.315924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:49.316099 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:49.316223 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:49.316422 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.317260 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318248 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:49.318543 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:49.318629 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:49.318944 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:49.319309 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:49.319404 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:49.319667 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 16:42:49.319726 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":81,"threads_started":1}
I20260925 16:42:49.330396 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22637
I20260925 16:42:49.336834 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22712
I20260925 16:42:49.342563 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 22787
2026-09-25T16:42:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3085 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:42:49 dist-test-slave-kxl8 krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:42:49 dist-test-slave-kxl8 krb5kdc[23004](info): set up 2 sockets
Sep 25 16:42:49 dist-test-slave-kxl8 krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260925 16:42:50.377794 391 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/test-user.keytab.
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
Entry for principal kudu/127.0.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab.
I20260925 16:42:50.497331 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42829
--webserver_interface=127.0.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:42:50.607925 23017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:50.608245 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:50.608310 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:50.608387 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:50.613266 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:50.613355 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:50.613389 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:50.613430 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:50.613462 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:50.618678 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.254.keytab
--principal=kudu/127.0.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.254:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:50.620029 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:50.621367 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:42:50.631372 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:50.631384 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:50.631438 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:50.631420 23017 server_base.cc:1034] running on GCE node
I20260925 16:42:50.632109 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:50.633163 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:50.634335 23017 hybrid_clock.cc:648] HybridClock initialized: now 1790354570634316 us; error 31 us; skew 500 ppm
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:50.636970 23017 init.cc:377] Logged in from keytab as kudu/127.0.97.254@KRBTEST.COM (short username kudu)
I20260925 16:42:50.637840 23017 webserver.cc:460] Webserver started at http://127.0.97.254:40755/ using document root <none> and password file <none>
I20260925 16:42:50.638367 23017 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:50.638451 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:50.638665 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:50.640573 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data/instance:
uuid: "a2f8336024154becb8028aaa892af91a"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "92c22e62e270a9c8dcb55e8eaa30777c"
server_key_iv: "72cde35e0d0830125d6df31dfde94985"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.641119 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance:
uuid: "a2f8336024154becb8028aaa892af91a"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "92c22e62e270a9c8dcb55e8eaa30777c"
server_key_iv: "72cde35e0d0830125d6df31dfde94985"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.644639 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:42:50.646994 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:50.648185 23017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:42:50.648313 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal
uuid: "a2f8336024154becb8028aaa892af91a"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "92c22e62e270a9c8dcb55e8eaa30777c"
server_key_iv: "72cde35e0d0830125d6df31dfde94985"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.648418 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:50.671255 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:50.674773 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:50.674999 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:50.682168 23017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.254:42829
I20260925 16:42:50.682471 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.254:42829 every 8 connection(s)
I20260925 16:42:50.683320 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/data/info.pb
I20260925 16:42:50.686059 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:50.689633 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:50.693455 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 23017
I20260925 16:42:50.693585 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-0/wal/instance
I20260925 16:42:50.693878 391 external_mini_cluster.cc:1428] Setting key b8e80448c85a83e2f69f74a4801a5d56
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.253@KRBTEST.COM" created.
I20260925 16:42:50.705446 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.691878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42829 (local address 127.0.0.1:60336)
0925 16:42:50.692423 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.692476 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:50.693458 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:50.694233 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:50.694241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:50.694747 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:50.695122 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.695143 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.696241 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.696246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.697155 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.697164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.698083 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.698086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.698172 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.698805 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:50.698830 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:50.700499 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:50.702461 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.702471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.702480 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.702701 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.703013 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.703016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.703018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.703123 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.703748 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:50.703754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:50.703917 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 16:42:50.704113 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 16:42:50.704463 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":104,"spinlock_wait_cycles":76672,"thread_start_us":140,"threads_started":1}
I20260925 16:42:50.705446 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.691878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.97.254:42829)
0925 16:42:50.692384 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:50.692406 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:50.692605 (+ 199us) server_negotiation.cc:407] Connection header received
0925 16:42:50.693802 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:50.693806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:50.694037 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:50.694183 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:50.695281 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.696118 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.697291 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.697927 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.698120 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.700647 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:50.700652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:50.700661 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:50.700680 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:50.702361 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.702805 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.702815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.702817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.702910 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.703208 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.703215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.703216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.703547 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:50.703780 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:50.704186 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 16:42:50.704415 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":4864,"thread_start_us":140,"threads_started":1}
I20260925 16:42:50.707836 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:50.708485 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.708255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57194)
0925 16:42:50.708364 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.708432 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 16:42:50.708770 23035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
I20260925 16:42:50.709676 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.709470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57206)
0925 16:42:50.709588 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.709629 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 16:42:50.709897 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40643: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
Entry for principal kudu/127.0.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab.
I20260925 16:42:50.720045 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40643
--webserver_interface=127.0.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:42:50.759141 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } attempt: 1
I20260925 16:42:50.760046 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.759731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57208)
0925 16:42:50.759901 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.759965 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 16:42:50.760805 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.760612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57210)
0925 16:42:50.760711 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.760752 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:42:50.760968 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40643: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
W20260925 16:42:50.828088 23092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:50.828337 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:50.828361 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:50.828437 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 16:42:50.829150 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } attempt: 2
I20260925 16:42:50.829967 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.829690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57212)
0925 16:42:50.829834 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.829894 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 16:42:50.830602 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.830429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57222)
0925 16:42:50.830509 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.830555 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 16:42:50.830729 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.253:40643: Network error: Client connection negotiation failed: client connection to 127.0.97.253:40643: connect: Connection refused (error 111)
W20260925 16:42:50.832880 23092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:50.832930 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:50.832950 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:50.833005 23092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:50.833036 23092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:50.838305 23092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.253.keytab
--principal=kudu/127.0.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.253:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:50.839262 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:50.840219 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:50.848881 23092 server_base.cc:1034] running on GCE node
W20260925 16:42:50.848861 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:50.848863 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:50.848986 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:50.849623 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:50.850692 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:50.851866 23092 hybrid_clock.cc:648] HybridClock initialized: now 1790354570851832 us; error 48 us; skew 500 ppm
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:50.854692 23092 init.cc:377] Logged in from keytab as kudu/127.0.97.253@KRBTEST.COM (short username kudu)
I20260925 16:42:50.855551 23092 webserver.cc:460] Webserver started at http://127.0.97.253:37757/ using document root <none> and password file <none>
I20260925 16:42:50.856042 23092 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:50.856097 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:50.856251 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:50.858223 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data/instance:
uuid: "48fa41538ed6431fae69638cc1573657"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "a365c5403fa84a4d1d283a71d53cf76f"
server_key_iv: "18b78ec96245dfe7f8e4043a68928cbf"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.858711 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal/instance:
uuid: "48fa41538ed6431fae69638cc1573657"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "a365c5403fa84a4d1d283a71d53cf76f"
server_key_iv: "18b78ec96245dfe7f8e4043a68928cbf"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.862128 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:42:50.864300 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:50.865499 23092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:50.865607 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal
uuid: "48fa41538ed6431fae69638cc1573657"
format_stamp: "Formatted at 2026-09-25 16:42:50 on dist-test-slave-kxl8"
server_key: "a365c5403fa84a4d1d283a71d53cf76f"
server_key_iv: "18b78ec96245dfe7f8e4043a68928cbf"
server_key_version: "encryptionkey@0"
I20260925 16:42:50.865697 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:50.883086 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:50.886270 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:50.886468 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:50.892779 23092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.253:40643
I20260925 16:42:50.893077 23159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.253:40643 every 8 connection(s)
I20260925 16:42:50.893757 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/data/info.pb
I20260925 16:42:50.895807 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 23092
I20260925 16:42:50.895962 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-1/wal/instance
I20260925 16:42:50.896296 391 external_mini_cluster.cc:1428] Setting key 894fef6a158260673702105bff16dd45
I20260925 16:42:50.897223 23160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:50.901562 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.252@KRBTEST.COM" created.
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:50.918238 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.903541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.97.254:42829)
0925 16:42:50.903710 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:50.903717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:50.904594 (+ 877us) server_negotiation.cc:407] Connection header received
0925 16:42:50.905716 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:50.905721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:50.905804 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:50.905894 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:50.907600 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.908381 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.910497 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.910801 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.910886 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.913957 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:50.913961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:50.913964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:50.913997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:50.915880 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.916479 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.916482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.916483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.916609 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.916987 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.916989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.916990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.917169 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:50.917249 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:50.917944 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 16:42:50.918096 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:42:50.919107 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.903942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42829 (local address 127.0.0.1:60344)
0925 16:42:50.904431 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.904485 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:50.905500 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:50.906148 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:50.906157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:50.906832 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:50.907406 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.907428 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.908538 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.908545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.910334 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.910361 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.910984 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.910990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.911224 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.911902 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:50.911931 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:50.913770 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:50.916017 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.916022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.916035 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.916328 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.916709 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.916711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.916713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.916896 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.917317 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:50.917326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:50.917619 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 16:42:50.917833 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 16:42:50.918131 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260925 16:42:50.920522 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:50.921896 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
Entry for principal kudu/127.0.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab.
I20260925 16:42:50.924612 391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
/tmp/dist-test-taskVEDvT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:37937
--webserver_interface=127.0.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
Sep 25 16:42:50 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:42:50.932086 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.921018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57236)
0925 16:42:50.921254 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.921333 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:50.921476 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:50.921935 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:50.921938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:50.922150 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:50.922248 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.922274 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.923578 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.923586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.924461 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.924471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.925307 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.925311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.925424 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.925899 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:50.925910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:50.928200 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:50.930351 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.930354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.930356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.930614 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.930881 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.930886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.930888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.930984 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.931563 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:50.931566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:50.931669 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 16:42:50.931774 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:42:50.931891 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2}
I20260925 16:42:50.932477 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.922280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57240)
0925 16:42:50.922377 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.922390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:50.922458 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:50.922843 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:50.922846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:50.923009 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:50.923084 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.923089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.924004 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.924008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.924524 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:50.924551 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.925412 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.925419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:50.925577 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.926160 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:50.926182 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:50.927840 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:50.930999 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.931002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.931004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.931211 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.931478 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:50.931482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:50.931484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:50.931578 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:50.931972 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:50.931975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:50.932063 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 16:42:50.932146 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:42:50.932290 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 16:42:50.932718 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.921115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.0.97.253:40643)
0925 16:42:50.921453 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:50.921462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:50.921482 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:42:50.921550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:50.921554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:50.921774 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:50.921921 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:50.922464 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.923459 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.924601 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.925178 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.925309 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.928339 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:50.928342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:50.928344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:50.928367 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:50.930236 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.930715 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.930721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.930722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.930795 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.931071 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.931077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.931081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.931457 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:50.931600 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:50.931888 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 16:42:50.932066 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":253,"spinlock_wait_cycles":896,"thread_start_us":129,"threads_started":1}
I20260925 16:42:50.932996 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.922344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.253:40643)
0925 16:42:50.922659 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:50.922665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:50.922675 (+ 10us) server_negotiation.cc:407] Connection header received
0925 16:42:50.922699 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:50.922701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:50.922737 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:50.922828 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:50.923188 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.923891 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.924644 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:50.925233 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:50.925805 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:50.928044 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:50.928057 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:50.928074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:50.928104 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:50.930890 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.931301 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.931306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.931308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.931394 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:50.931679 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:50.931682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:50.931683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:50.931866 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:50.932663 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:50.932733 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 16:42:50.932851 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260925 16:42:50.934486 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:50.934954 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:42:50.935192 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.934945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51302)
0925 16:42:50.935068 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.935138 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:42:50.935369 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
I20260925 16:42:50.935724 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.935465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51304)
0925 16:42:50.935606 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.935659 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 16:42:50.936312 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.936125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51308)
0925 16:42:50.936219 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.936261 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 16:42:50.936712 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.936470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51320)
0925 16:42:50.936605 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.936652 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 16:42:50.936677 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
W20260925 16:42:50.936841 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
I20260925 16:42:50.968024 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 1
I20260925 16:42:50.969026 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.968694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51322)
0925 16:42:50.968870 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.968934 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 16:42:50.969841 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.969567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51338)
0925 16:42:50.969689 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.969739 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:42:50.969985 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
I20260925 16:42:50.985915 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 1
I20260925 16:42:50.986783 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.986417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51340)
0925 16:42:50.986592 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.986673 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 16:42:50.987483 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:50.987255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51344)
0925 16:42:50.987355 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:50.987418 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 16:42:50.987633 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
W20260925 16:42:51.036459 23167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:42:51.036762 23167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:42:51.036787 23167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:42:51.036823 23167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:42:51.040828 23167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:42:51.040887 23167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:42:51.040910 23167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:42:51.040928 23167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:42:51.040943 23167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:42:51.045157 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 2
I20260925 16:42:51.045938 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.045666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51352)
0925 16:42:51.045800 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.045864 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 16:42:51.046312 23167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.254:42829,127.0.97.253:40643,127.0.97.252:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/krb5kdc/kudu_127.0.97.252.keytab
--principal=kudu/127.0.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.252:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:42:51.046700 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.046436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51358)
0925 16:42:51.046616 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.046653 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 16:42:51.046864 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
I20260925 16:42:51.047386 23167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:42:51.048346 23167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:42:51.055796 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 2
I20260925 16:42:51.056615 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.056285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51372)
0925 16:42:51.056417 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.056472 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 16:42:51.056731 23174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:42:51.056721 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:51.057065 23167 server_base.cc:1034] running on GCE node
W20260925 16:42:51.057194 23173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:42:51.057463 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.057239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51382)
0925 16:42:51.057355 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.057401 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:42:51.057602 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.252:37937: Network error: Client connection negotiation failed: client connection to 127.0.97.252:37937: connect: Connection refused (error 111)
I20260925 16:42:51.058117 23167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:42:51.059249 23167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:42:51.060427 23167 hybrid_clock.cc:648] HybridClock initialized: now 1790354571060390 us; error 47 us; skew 500 ppm
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:42:51.063107 23167 init.cc:377] Logged in from keytab as kudu/127.0.97.252@KRBTEST.COM (short username kudu)
I20260925 16:42:51.063982 23167 webserver.cc:460] Webserver started at http://127.0.97.252:41001/ using document root <none> and password file <none>
I20260925 16:42:51.064563 23167 fs_manager.cc:359] Metadata directory not provided
I20260925 16:42:51.064649 23167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:42:51.064867 23167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:42:51.066936 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data/instance:
uuid: "e5dc06f681c74f9b9293848fd6cbf1f6"
format_stamp: "Formatted at 2026-09-25 16:42:51 on dist-test-slave-kxl8"
server_key: "79d4b165dffbfa69910c6ca908fc1559"
server_key_iv: "fb1d4e5ba7c4f6a096cf1f453fef1c89"
server_key_version: "encryptionkey@0"
I20260925 16:42:51.067502 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal/instance:
uuid: "e5dc06f681c74f9b9293848fd6cbf1f6"
format_stamp: "Formatted at 2026-09-25 16:42:51 on dist-test-slave-kxl8"
server_key: "79d4b165dffbfa69910c6ca908fc1559"
server_key_iv: "fb1d4e5ba7c4f6a096cf1f453fef1c89"
server_key_version: "encryptionkey@0"
I20260925 16:42:51.071513 23167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 16:42:51.073871 23183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:42:51.075038 23167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:42:51.075136 23167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data,/tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal
uuid: "e5dc06f681c74f9b9293848fd6cbf1f6"
format_stamp: "Formatted at 2026-09-25 16:42:51 on dist-test-slave-kxl8"
server_key: "79d4b165dffbfa69910c6ca908fc1559"
server_key_iv: "fb1d4e5ba7c4f6a096cf1f453fef1c89"
server_key_version: "encryptionkey@0"
I20260925 16:42:51.075222 23167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:42:51.096092 23167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:42:51.099792 23167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:42:51.100005 23167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:42:51.107225 23167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.252:37937
I20260925 16:42:51.107489 23235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.252:37937 every 8 connection(s)
I20260925 16:42:51.108246 23167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/data/info.pb
I20260925 16:42:51.111050 23236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:42:51.112381 391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu as pid 23167
I20260925 16:42:51.112511 391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVEDvT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790354288967115-391-0/minicluster-data/master-2/wal/instance
I20260925 16:42:51.112844 391 external_mini_cluster.cc:1428] Setting key 53fe9b4ff5d1d043bb26468322d63f73
I20260925 16:42:51.115628 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:51.126742 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.117468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.97.254:42829)
0925 16:42:51.117609 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.117614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.117938 (+ 324us) server_negotiation.cc:407] Connection header received
0925 16:42:51.118071 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.118077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.118128 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.118216 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.119000 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.119634 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.120723 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.120923 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.120995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.123450 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.123453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.123455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.123475 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.124949 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.125457 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.125461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.125463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.125544 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.125813 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.125816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.125818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.125991 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.126092 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.126460 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.126579 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:42:51.127213 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.117416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.254:42829 (local address 127.0.0.1:60352)
0925 16:42:51.117812 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.117859 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:51.117975 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:51.118298 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:51.118303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:51.118739 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:51.118878 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.118884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.119754 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.119756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.120597 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.120608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.121079 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.121086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.121232 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.121785 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:51.121801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:51.123328 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:51.125074 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.125082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.125087 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.125359 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.125630 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.125635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.125637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.125721 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.126091 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:51.126098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:51.126226 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 16:42:51.126363 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 16:42:51.126504 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":176,"threads_started":1}
I20260925 16:42:51.127213 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.114172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.97.252:37937)
0925 16:42:51.114567 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.114586 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.114630 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:42:51.115442 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.115464 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.115895 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.116251 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.116895 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.118174 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.119259 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.119902 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.120009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.122584 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.122591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.122607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.122627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.124432 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.124840 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.124846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.124851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.124952 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.125220 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.125225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.125227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.125644 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.125867 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.126115 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.126388 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260925 16:42:51.128309 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.254@KRBTEST.COM
I20260925 16:42:51.136345 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.128775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.253:40643 (local address 127.0.0.1:57252)
0925 16:42:51.128897 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.128911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:51.128988 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:51.129244 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:51.129247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:51.129437 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:51.129547 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.129554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.130436 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.130439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.130949 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.130957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.131380 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.131384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.131475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.131907 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:51.131919 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:51.133069 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:51.134963 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.134966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.134968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.135155 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.135438 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.135440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.135442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.135519 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.135862 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:51.135865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:51.135974 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 16:42:51.136074 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:42:51.136192 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 16:42:51.136373 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.128886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.0.97.253:40643)
0925 16:42:51.129030 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.129035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.129048 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:42:51.129092 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.129095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.129138 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.129232 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.129674 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.130309 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.131053 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.131276 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.131350 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.133199 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.133202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.133204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.133224 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.134846 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.135249 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.135254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.135256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.135344 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.135605 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.135610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.135611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.135775 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.135869 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.136136 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.136240 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:42:51.137260 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.129514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.97.254:42829)
0925 16:42:51.129638 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.129642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.129655 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:42:51.129755 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.129757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.129797 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.129878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.130402 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.130861 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.131792 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.132018 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.132118 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.134289 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.134293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.134295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.134318 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.135708 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.136118 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.136123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.136125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.136198 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.136491 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.136526 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.136527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.136679 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.136898 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.137007 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.137111 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:42:51.137476 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.252@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.253@KRBTEST.COM
I20260925 16:42:51.145567 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.137919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51412)
0925 16:42:51.138204 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.138221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:51.138299 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:51.138521 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:51.138526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:51.138681 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:51.138774 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.138780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.139703 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.139707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.140369 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.140377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.140917 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.140919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.141017 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.141508 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:51.141527 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:51.142598 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:51.144217 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.144223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.144225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.144403 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.144648 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.144652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.144654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.144729 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.145062 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:51.145064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:51.145138 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 16:42:51.145239 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 16:42:51.145382 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260925 16:42:51.145567 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.137999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.0.97.252:37937)
0925 16:42:51.138094 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.138097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.138279 (+ 182us) server_negotiation.cc:407] Connection header received
0925 16:42:51.138390 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.138396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.138436 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.138508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.138881 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.139613 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.140465 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.140818 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.140935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.142727 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.142730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.142746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.142772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.144122 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.144489 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.144494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.144496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.144565 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.144810 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.144815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.144817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.144966 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.145071 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.145313 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.145435 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":34}
I20260925 16:42:51.146028 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 3
I20260925 16:42:51.147382 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.138366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.0.97.253:40643)
0925 16:42:51.138464 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.138467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.138518 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:42:51.138625 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.138630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.138675 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.138760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.139449 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.140072 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.140979 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.141336 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.141426 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.143884 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.143886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.143889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.143909 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.145234 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.145741 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.145746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.145749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.145842 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.146128 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.146132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.146134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.146410 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.146775 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.147109 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.147223 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:42:51.148813 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } attempt: 3
I20260925 16:42:51.149402 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: Bootstrap starting.
I20260925 16:42:51.150839 391 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.254@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:42:51.152341 23236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:51.153240 23236 log.cc:824] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:51.152098 23059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a2f8336024154becb8028aaa892af91a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368
Sep 25 16:42:51 dist-test-slave-kxl8 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.253@KRBTEST.COM for kudu/127.0.97.252@KRBTEST.COM
I20260925 16:42:51.156008 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: No bootstrap required, opened a new log
I20260925 16:42:51.156082 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.146427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51418)
0925 16:42:51.146556 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.146571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:51.146645 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:51.146966 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:51.146969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:51.147170 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:51.147277 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.147287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.148170 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.148177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.148688 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.148696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.149798 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.149802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.149896 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.150377 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:51.150393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:51.151805 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:51.154079 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.154083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.154085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.154336 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.154635 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.154638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.154640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.154741 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.155249 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:51.155255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:51.155381 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 16:42:51.155748 (+ 367us) client_negotiation.cc:240] Negotiation successful
0925 16:42:51.155914 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 16:42:51.156229 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.146651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.0.97.252:37937)
0925 16:42:51.146744 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.146748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.146760 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:42:51.146793 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.146799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.146836 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.147388 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.147474 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.148044 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.148802 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.149669 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.151954 (+ 2285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.152073 (+ 119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.152076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.152078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.152099 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.153965 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.154456 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.154460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.154461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.154556 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.154861 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.154864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.154866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.155100 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.155228 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.155864 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.156041 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:42:51.158870 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.149166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.252:37937 (local address 127.0.0.1:51434)
0925 16:42:51.149344 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 16:42:51.149359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:42:51.149431 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:42:51.149889 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:42:51.149892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:42:51.150068 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:42:51.150151 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.150157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.151203 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.151206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.151946 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:42:51.151956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.152618 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.152623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:42:51.152698 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.153166 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:42:51.153179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:42:51.154456 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:42:51.157324 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.157327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.157329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.157583 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.157876 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:42:51.157879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:42:51.157880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:42:51.157970 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:42:51.158382 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:42:51.158386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:42:51.158480 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 16:42:51.158585 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:42:51.158714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:42:51.159096 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.149397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.0.97.252:37937)
0925 16:42:51.149676 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.149684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.149696 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:42:51.149724 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.149727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.149762 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.150255 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.150333 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.151082 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.152131 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.152476 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.152873 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.154614 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.154620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.154623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.154661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.157209 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.157688 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.157692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.157693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.157782 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.158076 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.158079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.158081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.158278 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.158380 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.158676 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.158783 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260925 16:42:51.159528 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: Bootstrap starting.
I20260925 16:42:51.160087 23236 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.160300 23236 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dc06f681c74f9b9293848fd6cbf1f6, State: Initialized, Role: FOLLOWER
I20260925 16:42:51.160614 23236 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.161104 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.161232 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.161748 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:51.161849 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: Bootstrap starting.
I20260925 16:42:51.161968 23236 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:51.162358 23085 log.cc:824] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:42:51.164367 23160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: Neither blocks nor log segments found. Creating new log.
I20260925 16:42:51.164590 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: No bootstrap required, opened a new log
I20260925 16:42:51.165218 23160 log.cc:824] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: Log is configured to *not* fsync() on all Append() calls
W20260925 16:42:51.166996 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:51.167168 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:51.167443 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: No bootstrap required, opened a new log
I20260925 16:42:51.168653 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.168938 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f8336024154becb8028aaa892af91a, State: Initialized, Role: FOLLOWER
I20260925 16:42:51.165453 23167 master_runner.cc:186] Error getting master registration for 127.0.97.254:42829: OK, instance_id { permanent_uuid: "a2f8336024154becb8028aaa892af91a" instance_seqno: 1790354570680194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:42:51.169240 23167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:42:51.169312 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.169811 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.169955 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.170706 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:51.171110 23160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.171270 23160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fa41538ed6431fae69638cc1573657, State: Initialized, Role: FOLLOWER
I20260925 16:42:51.171610 23160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.172072 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.172196 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.172880 23160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:42:51.173391 23059 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:42:51.173512 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:51.174788 23059 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.175683 23059 leader_election.cc:288] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [CANDIDATE]: Term 1 election: Requested vote from peers 48fa41538ed6431fae69638cc1573657 (127.0.97.253:40643), e5dc06f681c74f9b9293848fd6cbf1f6 (127.0.97.252:37937)
W20260925 16:42:51.175679 23270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:51.175796 23270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:51.176966 23134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2f8336024154becb8028aaa892af91a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48fa41538ed6431fae69638cc1573657"
I20260925 16:42:51.177188 23134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:51.178365 23134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f8336024154becb8028aaa892af91a in term 1.
I20260925 16:42:51.178807 23035 leader_election.cc:302] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fa41538ed6431fae69638cc1573657, a2f8336024154becb8028aaa892af91a; no voters:
I20260925 16:42:51.178839 23210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2f8336024154becb8028aaa892af91a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6"
I20260925 16:42:51.179056 23210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:42:51.179123 23258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:42:51.179332 23258 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [term 1 LEADER]: Becoming Leader. State: Replica: a2f8336024154becb8028aaa892af91a, State: Running, Role: LEADER
I20260925 16:42:51.179697 23258 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } }
I20260925 16:42:51.180850 23210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f8336024154becb8028aaa892af91a in term 1.
W20260925 16:42:51.181140 23038 tablet.cc:2369] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:42:51.181469 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f8336024154becb8028aaa892af91a. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.181578 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:51.183544 23017 master_runner.cc:186] Error getting master registration for 127.0.97.253:40643: OK, instance_id { permanent_uuid: "48fa41538ed6431fae69638cc1573657" instance_seqno: 1790354570891060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:42:51.187088 23017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:42:51.191335 23278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:42:51.192066 23278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 16:42:51.194720 23289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:42:51.194801 23289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:42:51.196939 23134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [term 1 FOLLOWER]: Refusing update from remote peer a2f8336024154becb8028aaa892af91a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:51.196983 23210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [term 1 FOLLOWER]: Refusing update from remote peer a2f8336024154becb8028aaa892af91a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:42:51.197597 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [LEADER]: Connected to new peer: Peer: permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:51.197768 23258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:42:51.201313 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f8336024154becb8028aaa892af91a. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.201465 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.201767 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f8336024154becb8028aaa892af91a. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.201879 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.206043 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.206208 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:51.207369 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.207484 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.207791 23291 mvcc.cc:205] Tried to move back new op lower bound from 7333292323615510528 to 7333292323555602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333292323615510528}]
I20260925 16:42:51.208150 23294 mvcc.cc:205] Tried to move back new op lower bound from 7333292323615510528 to 7333292323555602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333292323615510528}]
I20260925 16:42:51.218532 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.202147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.97.252:37937)
0925 16:42:51.202252 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.202257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.208509 (+ 6252us) server_negotiation.cc:407] Connection header received
0925 16:42:51.208712 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.208716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.208769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.208823 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.209904 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.210731 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.211827 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.212173 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.212245 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.214197 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.214201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.214204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.214231 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.216659 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.217151 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.217155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.217157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.217242 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.217574 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.217577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.217578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.217791 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.217912 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.218243 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.218365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 16:42:51.220284 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.220420 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a [sys.catalog]: This master's current role is: LEADER
I20260925 16:42:51.221737 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2f8336024154becb8028aaa892af91a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f8336024154becb8028aaa892af91a" member_type: VOTER last_known_addr { host: "127.0.97.254" port: 42829 } } peers { permanent_uuid: "48fa41538ed6431fae69638cc1573657" member_type: VOTER last_known_addr { host: "127.0.97.253" port: 40643 } } peers { permanent_uuid: "e5dc06f681c74f9b9293848fd6cbf1f6" member_type: VOTER last_known_addr { host: "127.0.97.252" port: 37937 } } }
I20260925 16:42:51.221845 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:42:51.222548 23293 mvcc.cc:205] Tried to move back new op lower bound from 7333292323615510528 to 7333292323555602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333292323615510528}]
I20260925 16:42:51.223194 23278 catalog_manager.cc:1347] Generated new cluster ID: 843db7c3bcb64e1e8a6c46f515487ae3
I20260925 16:42:51.223264 23278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:42:51.225889 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.202584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.254:42829)
0925 16:42:51.202706 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.202710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.212711 (+ 10001us) server_negotiation.cc:407] Connection header received
0925 16:42:51.212882 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.212886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.212950 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.213018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.215017 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.216131 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.220124 (+ 3993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.220487 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.220608 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.222227 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.222233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.222237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.222264 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.224296 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.224738 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.224741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.224743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.224818 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.225057 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.225060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.225061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.225237 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.225348 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.225642 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.225750 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:42:51.231662 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:51.202213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.97.253:40643)
0925 16:42:51.202323 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.202328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.209533 (+ 7205us) server_negotiation.cc:407] Connection header received
0925 16:42:51.209712 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.209715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.209761 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.209812 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.210712 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.211491 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.213952 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:51.214286 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:51.214361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:51.222071 (+ 7710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:42:51.222074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:42:51.222077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:42:51.222109 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:42:51.229871 (+ 7762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.230355 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.230358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.230360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.230451 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:42:51.230777 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:42:51.230782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:42:51.230784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:42:51.231005 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:42:51.231108 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:51.231402 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 16:42:51.231521 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:42:51.265365 23278 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:42:51.266685 23278 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:42:51.285301 23278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f8336024154becb8028aaa892af91a: Generated new TSK 0
I20260925 16:42:51.286213 23278 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:42:51.319872 23241 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.316944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.0.97.252:37937)
0925 16:42:51.317068 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.317074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.318844 (+ 1770us) server_negotiation.cc:407] Connection header received
0925 16:42:51.319048 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.319052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.319103 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.319156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.319748 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51456: BlockingRecv error: recv got EOF from 127.0.0.1:51456 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 16:42:51.321728 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.317646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.0.97.253:40643)
0925 16:42:51.317756 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.317760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.320332 (+ 2572us) server_negotiation.cc:407] Connection header received
0925 16:42:51.320761 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.320765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.320812 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.320861 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.321609 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57280: BlockingRecv error: recv got EOF from 127.0.0.1:57280 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 16:42:51.322438 23087 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.318290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.254:42829)
0925 16:42:51.318399 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.318403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.320517 (+ 2114us) server_negotiation.cc:407] Connection header received
0925 16:42:51.320988 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.320991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.321051 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.321101 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.322330 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60392: BlockingRecv error: recv got EOF from 127.0.0.1:60392 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 16:42:51.326681 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.325189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.97.253:40643)
0925 16:42:51.325299 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.325303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.325802 (+ 499us) server_negotiation.cc:407] Connection header received
0925 16:42:51.326015 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.326018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.326087 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.326137 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.326590 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57286: BlockingRecv error: recv got EOF from 127.0.0.1:57286 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:42:51.331254 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.329841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.97.253:40643)
0925 16:42:51.329934 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.329938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.330451 (+ 513us) server_negotiation.cc:407] Connection header received
0925 16:42:51.330618 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.330621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.330667 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.330712 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.331165 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57290: BlockingRecv error: recv got EOF from 127.0.0.1:57290 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 16:42:51.336406 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.335041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.97.253:40643)
0925 16:42:51.335130 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.335134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.335599 (+ 465us) server_negotiation.cc:407] Connection header received
0925 16:42:51.335765 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.335768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.335813 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.335859 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.336314 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:42:51.343778 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.342314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.97.253:40643)
0925 16:42:51.342434 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.342438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.342907 (+ 469us) server_negotiation.cc:407] Connection header received
0925 16:42:51.343114 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.343117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.343164 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.343210 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.343685 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57314: BlockingRecv error: recv got EOF from 127.0.0.1:57314 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 16:42:51.356033 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.354292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.253:40643)
0925 16:42:51.354406 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.354411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.354967 (+ 556us) server_negotiation.cc:407] Connection header received
0925 16:42:51.355185 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.355188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.355238 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.355293 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.355874 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57322: BlockingRecv error: recv got EOF from 127.0.0.1:57322 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 16:42:51.376266 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.374522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.0.97.253:40643)
0925 16:42:51.374646 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.374651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.375227 (+ 576us) server_negotiation.cc:407] Connection header received
0925 16:42:51.375449 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.375453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.375511 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.375570 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.376150 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57338: BlockingRecv error: recv got EOF from 127.0.0.1:57338 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:42:51.411949 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.410785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.0.97.253:40643)
0925 16:42:51.410901 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.410905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.411115 (+ 210us) server_negotiation.cc:407] Connection header received
0925 16:42:51.411232 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.411234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.411280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.411371 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.411829 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57340: BlockingRecv error: recv got EOF from 127.0.0.1:57340 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 16:42:51.479436 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.478157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.0.97.253:40643)
0925 16:42:51.478314 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.478319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.478533 (+ 214us) server_negotiation.cc:407] Connection header received
0925 16:42:51.478689 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.478697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.478744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.478831 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.479319 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57354: BlockingRecv error: recv got EOF from 127.0.0.1:57354 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:42:51.610913 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.609717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.0.97.253:40643)
0925 16:42:51.609871 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.609875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.610025 (+ 150us) server_negotiation.cc:407] Connection header received
0925 16:42:51.610206 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.610209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.610256 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.610345 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.610798 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57364: BlockingRecv error: recv got EOF from 127.0.0.1:57364 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 16:42:51.870115 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:42:51.868894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.97.253:40643)
0925 16:42:51.869062 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:51.869067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:51.869204 (+ 137us) server_negotiation.cc:407] Connection header received
0925 16:42:51.869388 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:51.869396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:51.869442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:51.869529 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:42:51.869988 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57378: BlockingRecv error: recv got EOF from 127.0.0.1:57378 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:42:52.168597 23255 catalog_manager.cc:1259] Loaded cluster ID: 843db7c3bcb64e1e8a6c46f515487ae3
I20260925 16:42:52.168676 23255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: loading cluster ID for follower catalog manager: success
I20260925 16:42:52.170514 23255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: acquiring CA information for follower catalog manager: success
I20260925 16:42:52.171728 23255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5dc06f681c74f9b9293848fd6cbf1f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:42:52.196017 23289 catalog_manager.cc:1259] Loaded cluster ID: 843db7c3bcb64e1e8a6c46f515487ae3
I20260925 16:42:52.196125 23289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: loading cluster ID for follower catalog manager: success
I20260925 16:42:52.197785 23289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: acquiring CA information for follower catalog manager: success
I20260925 16:42:52.198910 23289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48fa41538ed6431fae69638cc1573657: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:42:52.387599 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:52.383997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.97.253:40643)
0925 16:42:52.384290 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:52.384293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:52.384329 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:42:52.384466 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:52.384469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:52.384598 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:52.384695 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:52.384902 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.385448 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.386233 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.386444 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.386517 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:52.386782 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:52.387208 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:52.387296 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:52.387342 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 16:42:52.387386 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260925 16:42:52.393898 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:52.390577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.97.252:37937)
0925 16:42:52.390792 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:52.390795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:52.390807 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:42:52.390880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:52.390882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:52.390991 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:52.391098 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:52.391267 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.391777 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.392527 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.392742 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.392817 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:52.393060 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:52.393527 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:52.393619 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:52.393669 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:42:52.393713 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":64,"threads_started":1}
I20260925 16:42:52.400269 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:52.396533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.0.97.253:40643)
0925 16:42:52.396725 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:52.396731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:52.396947 (+ 216us) server_negotiation.cc:407] Connection header received
0925 16:42:52.397145 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:52.397148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:52.397198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:52.397235 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:52.397672 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.398238 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.399111 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.399395 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.399487 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:52.399798 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:52.399876 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:52.399954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:52.400065 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 16:42:52.400116 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 16:42:52.400524 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:52.396943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.0.97.252:37937)
0925 16:42:52.397053 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:52.397056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:52.397305 (+ 249us) server_negotiation.cc:407] Connection header received
0925 16:42:52.397447 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:52.397450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:52.397507 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:52.397544 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:52.397865 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.398371 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.399455 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.399687 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.399789 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:52.400040 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:52.400106 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:52.400173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:52.400333 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 16:42:52.400378 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:42:52.401945 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:42:52.396528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.97.254:42829)
0925 16:42:52.396755 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 16:42:52.396760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:42:52.396832 (+ 72us) server_negotiation.cc:407] Connection header received
0925 16:42:52.397042 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:42:52.397045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:42:52.397206 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:42:52.397388 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:42:52.398350 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.399160 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.400240 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:42:52.400708 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:42:52.400801 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:42:52.401312 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:42:52.401578 (+ 266us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:42:52.401650 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:42:52.401727 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:42:52.401772 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260925 16:42:52.404613 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 23017
I20260925 16:42:52.411116 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 23092
I20260925 16:42:52.417014 391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVEDvT3/build/debug/bin/kudu with pid 23167
2026-09-25T16:42:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3074 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6159 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283440 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST